Xtensive – News Blog

Help Server 2.1 is released

2009-07-07T14:19:00.006+06:00

New features:
- Custom TOC icons.
- Localization. Language is detected automatically, currently English and Russian are supported. If you'd like to provide its localization for your own language, please send us [InstallationFolder]\Web\WebResources\WebResources.xx.resx file for it.
- Configuration is fully based on Web.config file now.

Full list of changes is published in Revision History. Updated installers are already available in Downloads section.

ADO.NET Data Services (Astoria) sample for DO4

2009-07-02T21:21:00.009+06:00

Here it is: http://www.x-tensive.com/Downloads/Default.aspx?Path=DataObjects.Net\Samples

What's there:
- Simple Customer-Order model
- Astoria service sharing it
- WindowsForms client consuming it
- Silverlight client consuming it.

Silverlight client screenshots:

WindowsForms client screenshots:

Installation & running:

SilverlightClient.Web:

0. Install Silverlight Tools and Silverlight 2.0.

1. Create web site with IIS Manager, home directory must be SilverlightClient.Web
2. Add Default.aspx to default pages (Web site properties -> Documents page)
3. Set ASP.NET Version to 2.x.x (Web site properties -> ASP.NET page -> ASP.NET version)
4. Add MIME type with extension: .xap and MIME type: application/x-Silverlight-app
(Web site properties -> HTTP Header page -> MIME Types button)
5. Enable anonymous access, disable "Integrated Windows authentication"
(Web site properties -> Directory Security page -> Authentication and access control,
Edit button)
6. Grant NTFS read permission to Everyone group for SilverlightClient.Web folder
6. Restart IIS.
7. Open http://localhost/[YourWebSite] in browser

WinFormsClient:

1. Run Server
2. Run WinFormsClient.

Please ask any questions related to this sample in comments to this post.

DataObjects.Net v4.0.2 is released

2009-07-02T04:33:00.006+06:00

Links:
- Download it
- View list of implemented issues.

What's new:

1. Improved installer

I hope we've fixed the last "big bugs" there. The most annoying ones are:
- 228: "Add\remove programs" issue on installing both DO4 and Xtensive.MSBuildTasks
- 232: DO 4.0.1 installer doesn't update assemblies located in PostSharp directory
- 233: Projects created by project template are bound to specific installation path of DO4

Because of 228 & 232, the recommended upgrade path to v4.0.2 is:
- Uninstall DO4. If the item absents in "Add\Remove programs", just remove its folder C:\Program Files\X-tensive.com (or the installation path you've chosen).
- Uninstall Xtensive.MSBuildTasks, if you have installed it. If the item absents in "Add\Remove programs", just remove its folders from C:\Program Files\MSBuild and C:\Program Files\X-tensive.com (or the installation path you've chosen).
- Uninstall all other components previously required by DO4, including Unity, Parallel Extensions, MSBuild Community Tasks and PostSharp.
- Install new DO4. It will suggest to install just PoshSharp. Everything else is optional now; all Unity and Parallel Extensions assemblies are installed into GAC automatically.

Other changes include the following ones:
- Installer automatically detects & requires to uninstall old version of DO4.
- All required assemblies are now installed into GAC. If you're worried about this, there are .bat files allowing to get rid of them with ease.
- There are new project templates (Console, Model, UnitTests, WebApplication, WPF). But they're only for C# for now.
- New Build.bat files build new DO automatically performing all "before first build" steps. So it's really easy now to make its custom build.

Useful links:
- Full list of installer-related issues
- New installation instruction
- New "Building DataObjects.Net" instruction

2. LINQ

As you might remember, two weeks ago we didn't support 2 LINQ features:
- Group joins
- First\Single(OrDefault) in subqueries (btw, as far as I remember, Single in subqueries isn't supported in EF at all)

Both features are supported now. So now we're fully ready to compare our LINQ implementation with others - a set of articles about this will appear here soon.

Useful links:
- LINQ-related issues.

3. Breaking changes in attributes

We've refactored our mapping attributes once more. Now there are:
- Separate [Association] attribute for associations
- Separate [Mapping] attribute allowing to specify mapping names.
- No more [Entity] attribute - it was necessary just to specify mapping name, but now this is handled by a separate attribute.

Earlier their functions were distributed over [Field] and old abstract MappingAttribute.

We think new version is better: specific (and, actually, more rarely necessary) features require specific attributes.

4. Schema upgrade

We've added Change FieldType Hint. So schema upgrade hint set is ideal now ;)

5. Documentation

We're slowly updating it. As you may find, we restructured our wiki. Manual is organized in step-by-step studying fashion now. Among other new articles, there is new Schema upgrade article - check it out.

6. ADO.NET Data Services (Astoria) sample

We've implemented ADO.NET Data Services (Astoria) sample on DO4. We decided to publish it separately:
- It isn't really polished yet
- It depends on Silverlight Tools, so we must decide if this additional dependency is acceptable.

It shows an Astoria service sharing entities via RESTful API, as well as WindowsForms and Silverlight clients consuming this service, showing and allowing to change the entities it gets.

What does this mean? You can share DO4 Entities using ADO.NET Data Services, query the service from the client using LINQ, update the entities on the client and send back the changes. Since Astoria client operates on Silverlight as well, you can implement Silverlight client utilizing DO4 on the server.

Btw... We disappointed in Astoria client features. You should do lots of tasks manually there, including registering of new entities, changed associations and so on. From the point of usability it's much worse than what is offered by DO4. So in general, upcoming sync will be much more attractive option for DO4 users. But on the other hand, Astoria allows to implement really simple RESTful integration API with almost zero coding.

The sample will be available @ our downloads section today.

7. Bugfixes

We've got really good results here. Earlier I wrote there are just few failing tests from about 1000 tests for Storage. Imagine:
- About 600 tests are related to our LINQ implementation, and indirectly - RSE implementation.
- All the tests produce the same results - even on Memory storage. This means our RSE execution & optimization engine works as expected.

So the version we have now seems really stable. Good luck trying it ;)

June and summer discounts: join DO4 camp!

2009-06-18T11:00:00.008+06:00

We're ready to start DataObjects.Net promotional campaign, and as frequently happens, it starts from pricing. Read this post till the end - we're providing huge discounts in June, but expect your involvement in exchange.

Price & licensing policy changes

This summer prices:

- Personal license: 299 USD (no changes)

- Internal license: 495 USD (-100 USD)

- SMB license: 995 USD (-1000 USD)

- Enterprise license: please request a qoute.

- Upgrade subscription: 50% of the license cost. To order it, you must order the original license with DOUPGRADE discount coupon code. Note that it's applicable only if your exising subscription isn't expired - it expires in 1 year form the first purchase; each purchased upgrade subscription adds 1 more year.

- Support subscription: now included into the cost of license.

- Implementation assistance and general consulting: 320 USD/day.

Discounts in June

First of all, we provide 50% discount on almost anything related to DO4, including:

- Licenses: they cost 150/248/498 USD in June. Quote for Entrprise license is provided by request.

- Upgarde subscriptions: 90/149/299 USD in June (so here the discount is 40%)

- Implementation assistance and consulting: 160 USD/day! This price will be definitely left intact till the end of summer.

Coupon code: JDDO4 - right now it is already included into all order pages.

Huge discounts for the people helping us

The conditions listed in this section will remain intact at least till the end of summer.

1. Promotion help discount: additional 50%.

Applicable to new license purchases only. You promise us to statisfy one of the following requirements during 1 month after your purchase:

1) You start or join, and further - maintain 3 different discussions on such developer's web site or forum as stackoverflow.com (it must be for .NET developers, and be well-known either worldwide or in your country).

The duscussions must be related to DO4 - its particular or upcoming features. We don't expect it must be fully positive - it must be simply honest. And there should be links to our wiki, blog or DO4 section on our web site.

2) If you maintain a well-known blog (well-known - of course, relatively - e.g. in your city; ideally, there must be >100 subscribers), you can publish 2 posts related to DO4 in it. Again, with your honest opinion and with links to us.

Please send the links to such discussions or posts to us - at least, as a proof you accomplished your part of contract ;) We'll join the discussion, if it will be necessary.

If you already did something similar, please notify us, and if we'll confirm the conditions listed here are satisfied, feel free to use the coupon code for this dicount.

As you see, we provide additional 50% discount (= 75% now) in exchange for several hours of your help. The final prices with this discount are:

- Licenses: 75/124/244 USD in June, or 150/248/498 USD later. Coupon code:JDDO4POSTER. Note that you can use it right now, but this implies your agree with the above conditions.

2. Contributor licenses: free SMB \ Enterprise licenses with lifetime upgrade and support

We're ready to provide free licenses, if you'd like to help us to develop or promote the product. What can be done to get a free license:

a) Significantly help us to develop some part of DO - for example:

- SQL DOM / storage provider for the database we don't support yet, or some notable part of it

- Sample application (currently we'd prefer good samples for ASP.NET or ADO.NET data services / Silverlight) or some notable part of it

- Useful tool (anything you miss ;) ) or some notable part of it

- Some other part we're planning to implement.

Expected complexity: if it took more than 10 full-time days, and we accepted the result, we'll definitely provide such a license. But other cases are discussable - we're ready for any proposals like this.

Obviously, it's necessary to discuss your plan with us before starting the implementation.

b) Promote DO4 at your local .NET, ALT.NET or similar user group - by making 1.5 hour overview of it there. You can choose any particular part you want, but ideally it should be based on our videos or on our own presentation of DO (will appear in June).

c) Do anything you want comparable to a) or b) by its effect. In fact, we expect you'll spend ~ 1-2 weeks on this, and the result of this will be attractive for DO users.

Remarks

As you may find, our goal for this summer is to grow up the community around DO4. Prices and earnings don't really matter for us.

But you may ask:

Why are you asking to pay something at all? The reason is simple: paid money is one of factors making things precious for you. We expect they'll make our simple "promo agreements" more important for both sides.

Why now? Because we feel the product is fully ready to be shown to much wider audience. Right now it can do more than e.g. currently available version of Entity Framework. But I hope by the end of summer it will be capable to simply smash it by the set of its built-in features.

Ok, of course I fully understand it won't be easy to compete with EF. But imagine, if EF would be shipped by someone else rather than Microsoft. I admit it would be really hard to get a big niche on ORM scene with its design. Really, they're providing features almost identical to NHibernate, but packed into "Microsoft way" box. It's really hard to identify what's new there. I'd also say this is dangerous for NHibernate as well.

I feel we're combining something very good both from open source and commercial development by running this campaign. I hope it will help to involve the people. Our purely commercial promo campaign will follow shortly - we'll be spending almost 5 times more on advertisements starting from the next week (I'm waiting for completion of a set of vital changes on our web site & Wiki). But your help is more important - just rumors, posts and articles make such products really famous.

So I suggest you to join our community right now. Together we'll go forward much faster. Ask yourselves:

- What commercial ORM is as open as DO4? We'd say even non-commercial are less open. Open source, availability under GPL, public issue tracker, open plans, Wiki that can be edited by you, possibility to contribute.

- Do you expect EF will ever run on open source platform? DO4 will definitely run on Mono in observable future.

- Is there any chance of getting DO4 development stopped during the nearest years, if we were ready to spend 2.5 years on developing this new version?

- Does DO4 differ from competitors? Even the current feature set shows it is. Architecturally it's completely new. Such schema upgrade layer, support for index storages, built-in IMDB, integrated query engine, upcoming sync - who else have these features?

- What features are expected in near future? There are tons of them on the way. You may find the product you see now, although being solid, shows a huge foundation for upcoming features. And that's cool, because the baby is just 2 week old! Ok, I know, but the pregnancy was long ;)

- Are we devoted to DO4? You must feel it worth much more than just money for us. It is a part of our image and reputation.

- Can you rely on it and on us? I hope, yes - taking into account above, as well as the experience (including support incidents) we've got during previous years of work on 1.X-3.X.

Join DO4 camp!

P.S. Even if you already have a license, think about writing something good about us ;)

[The post is copied from DataObjects.Net Team Blog]

Help Server 2.0.5 is out

2009-06-15T15:12:00.002+06:00

GZip compression and Google Analytics-like trackers support are among other new features. Full list of changes is published in Revision History.

Updated installers are already available in Downloads section.

DataObjects.Net v4.0.1 is out

2009-06-15T15:09:00.003+06:00

It is purely a bugfixing release. Full list of fixes we've made is available in DataObjects.Net issue tracker.

Updated installers are published in Downloads section.

P.S. We recommend you to subscribe to DataObjects.Net Team Blog - now it is updated almost daily.

DataObjects.Net v4.0 final is released

2009-06-09T05:56:00.013+06:00

DataObjects.Net v4.0 final is published in downloads section of our web site.

Most important changes we've made last month include:

1. Schema upgrade API. Shortly it will be fully described @ wiki, for now you can try out Upgrade Sample. There are 3 applications representing different versions of the same one. They must be run sequentially except the first one (it recreates the database), although you can try running them in order.

Upgrade API is really simple. To gracefully handle an upgrade, you must:

- Add AssemblyInfo attribute to AssemblyInfo.cs file: [assembly: AssemblyInfo("MyAssembly", "2.0")]

- Extend UpgradeHandler in each changed model assembly - it will assist to upgrade its structures.

- If necessary, override its AddUpgradeHints method and add instances of UpgradeHint ancestors there: RenameFieldHint, CopyFieldHint, RenameTypeHint. Note that upgrade hints are model-level hints, not schema-level.

- If necessary, apply [Recycled] attribute to some of your types and fields and override it OnUpgrade method to implement custom data migration logic from recycled structures there.

All these steps are shown in upgrade sample.

As you may find, UpgradeHandler offers more extension points, and there are schema-level upgrade hints as well. Normally they shouldn't be used - we translate our model-level hints to schema-level hints automatically. But some of them can be useful - for example, IgnoreHint instructing schema comparison layer to skip the specified schema object.

2. Refactored LINQ translator. Almost no differences outside (it passes just few more new tests), but a huge set of them inside. Faster, much simpler and eaisier to extend. Did I mention we're going to support custom LINQ extensions? Well, this version allows us to implement this gracefully.

3. Refactored mapping attributes. Since we've been shipping final, this became rather important. Details can be found here; new samples reflect this. Wiki will be updated in accordance shortly.

4. Improved installer & documentation

- Regular build installer now contains all downloadable prerequisites integrated

- VS.NET project template now contains much less references

- Brief version of class reference is provided @ help.x-tensive.com and installed as .Chm. Full version (.HxS) is integrated into VS.NET help collection.

Other important changes:
- Domain & Session events

- Improved PropertyConstraintAspect - now custom error message can be specified right in the attribute declaration

- New validators. See inheritors of PropertyConstraintAspect.

- We decided to get rid of VistaDB support. Earlier it was easy because of its good compatibility with SQL Server, but now it becomes more and more complex. There are actually many differences - normally acceptable for humans, but making much more complex machine translation for it. Our LINQ implementation and schema upgrade API provide very high level of compatibility between different RDBMS, and this compatibility costs a lot. So we'll focus on more frequently used RDBMS (Oracle, MySQL); furthermore, presence of our own memory provider and upcoming file system provider makes support of embedded databases much less necessary.

Public issue tracker is open

And finally, during the last month we've been using issue tracker on Google code (btw, it is really good). So all we did is easy to expose now: check out "Done" grid for v4.0 final. Previous changes are related to "Pre4.0" milestone, so they aren't shown here. Quicks statistics:

- 25 defects

- 14 new features

- 9 enchancements

- 9 refactorings

- 9 non-coding tasks.

Total: 66 issues.

If you're interested in near future, take a look at similar grid for v4.1. Some features mentioned here (mainly, fixes & tasks) will be separated to v4.0.5, which will be available in ~ 2 weeks.

And as I mentioned, from this point we're switching to monthly release cycle. So v4.1 can be expected in the first half of July.

Help Server 2.0 is out

2009-06-06T03:29:00.006+06:00

Please refer to Revision history for the complete list of changes (it's short, but all items are really important). You can download it here.

Note: we decreased help topic limit to 100 for Express Edition.

Its pages on our web site aren't updated yet (will be done next week), so there are no "Buy upgrade" links. But its User Guide is already updated, and all "Buy now" links are correct there.

P.S. I published first relatively big post @ blog.dataobjects.net.

Announcement: Help Server 2.0

2009-06-01T17:19:00.002+06:00

Hi everyone! We're glad to announce we're ready to release Help Server v2.0. This will happen on this week.

What's new:
- Full .HxS format support
- Lucene.Net based full-text search
- Faster startup. Full-text and keyword indexes are built in background now.
- Improved Index tab
- New URL-based query APIs: now you can invoke full-text search and index lookup using special URLs. Both these features are designed to integrate your class reference with conceptual description, manual or Wiki. Check out how index lookup works at wiki.dataobjects.net.

Note: help.x-tensive.com is running on one week old build. We're aware there are some issues. In particular:
- Many pages aren't covered by full-text search
- Index lookup doesn't open the queried page. It should do this, if there is just one candidate.

Certainly these issues will be resolved before release.

DataObjects.Net gets its own blog

2009-06-01T17:16:00.002+06:00

Go to http://blog.dataobjects.net. The detailed description is there now.

New video covering DO4 Console Sample

2009-05-24T21:35:00.005+06:00

Today we've published a new video covering Console Sample of DO4. It concentrates on such features as:
- Database schema building
- Metadata.Type, Metadata.Assembly and Metadata.Extension tables
- Batched key generation
- LINQ to SQL translation
- Running on built-in in-memory database (IMDB)
- LINQ to query execution plan translation (executable RSE providers).

Take a look at it - hopefully you'll enjoy it. Any comments are welcome.

The future of DO4

2009-05-24T20:54:00.011+06:00

After writing a short announcement related to the upcoming release, I understood I definitely should provide much more detailed information related to its future. So let's start:

Nearest plans

With v4.0 final we want to:
- Deliver a version that really includes all the basic features developers need.
- Show it is fully stable.
- Terminate the long-long chain of "[not yet working] v4.0 is updated [but is still not yet fully ready for production]" posts in this blog and get back to to the road of real releases.

And in addition, after releasing v4.0 final,
- We'll carefully plan each subsequent release. Its new features and delivery date will be precisely known.
- All the future updates will be logged in revision history. Actually we already switched to a public DataObjects.Net v4.0 issue tracker nearly one month ago.
- I expect further development will go faster. Luckily, now I don't see such big castles to storm as LINQ - at least, in observable future (6 months or so).

Btw, here is the answer on one of the frequent questions I heard during last months: are LINQ and schema upgarde really worth 9 months of work?

Yes.

I really hoped we reached this point in September-October, but was wrong. Frankly speaking, I really thought our internal RSE queries will be enough usable to start, or we'll be able to make them usable enough. But they were not, and all our ideas on how to make them better were looking pale in comparison with LINQ implementation. Moreover, it was supported by some of competitors. As you know, finally we've made a decision to get LINQ support ASAP. This was in the end of November. And from this moment we've been putting more and more resources on getting this done: initial implementation (certainly, quite lacky) made by a single person turned into ~ 4*4 man-months spent on the final implementation.

Comparison with some close competitors (LLBLGen Pro, NHibernate) shows this is definitely not the worst case we could get. Good LINQ implementation requires really lots of time, and for now only few products may claim they support LINQ well enough. In our case LINQ implementation was even more complex: as you might know, our LINQ to RSE translation layer produces real query plans, when it translates LINQ queries for index engines. For example, it automatically uses secondary indexes based on query criterias and statistics.

Another complex part (it costs about 20% of LINQ implementation) was upgrade, and v4.0 implements it really almost perfectly - now you can do all you want without SQL. To migrate your storage to the next version, you need just our upgrade hints and, optionally, your own "recycled" persistent types with all the features they integrate. This ensures your upgrade will be both simple and storage-independent. This is one of the most impressive advantages of new DO. I'm going to create a brief video overview of this feature, it will be available until the release.

Summer goals

#1: Number one RAD framework for database application development on .NET.

This summer we're going to show how to build full-featured WPF and ASP.NET applications (e.g. an issue tracker) on DataObjects.Net in 30 minutes or less.

DataObjects.Net, starting from its first version, was explicitly positioned as RAD tool for database application developers. Now we're going to show it really is such a RAD tool again - but certainly, much more powerful and much less limiting.

#2: Number one RAD framework for N-tier database application development.

The nearest feature we'll bring is integrated storage synchronization. It must dramatically simplify the development of N-tier database applications. Moreover, this summer we'll show file system provider for DataObjects.Net. In fact, a regular database implemented as our indexing engine. It will be initially positioned to be used mainly on the client side, but in future - who knows...

#3: Support the biggest part of RDBMS market.

We'll implement Oracle and MySQL support. Together with already supported RDBMS, they cover more than 90% of the market.

Media activities

This is one of our most annoying lacks. I understand we advertise ourselves quite bad from this point. Almost no articles, rather complex documentation and so on. Why?
a) Frankly speaking, until this moment we had nothing we could honestly write about. "We have really perfect ORM + IMDB, but unfortunately it can't be used in production because it destroys everything on any launch" - LOL, OMG and so on ;)
b) None of us, including me, are good article writers. At least in English.

Everything else seems a consequence of two above facts. No good articles on our web site = no articles on other sites = no exposure. But:
- v4.0 final fully eliminates a)
- b) can be eliminated only if we'll make ourselves to do this well.

If we'll really get b) resolved, we have a good chance to initiate a chain reaction. If not, it doesn't matter how cool is the tool we develop at all. It will sink unknown.

So what we'll do:
- Get #1 goal (full-featured application done in less than 30 minutes) implemented ASAP. This must be really impressive.
- Publish good video covering samples and features of DO4. I hope to finally get the production rate of our "home video" to one file per week at least.
- Eliminate most of the "red links" in our Wiki.
- Publish short articles covering interesting topics lying around ORM tools. For example, comparison of ORM framework performance, implemented LINQ features, discussion of verious features and approaches. Certainly, we'll be mostly on our own side. But if someone of our customers will decide to use the material to write and and publish an article e.g. at the CodeProject.com, we'll definitely favor this.

Announcement: DO4.0 final

2009-05-24T19:36:00.010+06:00

Hi everyone! We're glad to announce we've finally reached the point when we can fully honestly say we're ready to release DataObjects.Net v4.0 final. This will happen on the first week of this summer.

v4.0 final will include:
- All you can see in the currently available preview
- Almost ideal LINQ implementation
- Almost ideal schema upgrade API
- Providers for Microsoft SQL Server, PostgreSQL and our built-in in-memory database.

Everything else, including sync, additional providers, UI related tools, full-text indexing and search and so on will be released as the set of subsequent minor updates (v4.1, v4.2 and so on).

See the next post for further details.

LiveUI.net 0.4 release

2009-05-05T18:33:00.006+06:00

We are glad to announce first preview of LiveUI.net, which is our new product targeted to simplify web application development. This product is on early development stage now, but it is ready to get first impression. Its source code, live demo and documentation are available at http://liveui.net. For now it is absolutely free.

DataObjects.NET v4.0 got LINQ and upgrade API

2009-05-02T07:38:00.012+06:00

Only quick info for now, details will follow tomorrow:
- Almost complete LINQ support. Now only 12 tests out of almost 700 for LINQ are falling (~ about 15 are ignored), so we consider this part is 90% done. List of currently existing issues and lacks will be published shortly. See updated Console Sample - now it executes a bit more complex queries.
- First working version of our cool upgrade API. There are much more issues for now - e.g. it works only on SQL Server, understands results of our schema comparison layer only partially (for now it can't translate move or rename actions for columns and tables) and so on. But there is working Upgrade Sample utilizing this part of API, including IUpgradeHandler. We bet you dreamed about nearly this feature in v3.9 ;)
- We've started to work on our query optimizer for indexing storages. You can see what it already can do in Consile Sample - take a look on RSE queries it dumps on In-memory Storage, and you'll find we already automatically utilize secondary indexes! So if you even dreamed about storing indexed objects in memory and use LINQ to query them, we've just brought this to you. Obviously index access optimization isn't the only one we run - right now we support nearly the same optimizations as SQLite can do.
- There are some other new features, e.g. CachedQuery. To get some query cached, just wrap it into CachedQuery.Execute(() =>[YouOriginalQuery]). Btw, we implemented this pattern better than in EF - we don't require you to write a parameterized lambda here, and thus allow to use cached queries "in place".

That's all the brief description. Have a nice weekend!

P.S. Installers are already updated.

DataObjects.Net v4.0: status update

2009-04-02T15:10:00.008+06:00

Briefly, we're still fighting with schema evolution and LINQ support, but the end is near.

LINQ support

Currently we have the following set of issues that must be implemented to consider LINQ provider implementation project is finished:

Priority	Category	Implementation level	Task
1	Type mapping	LINQ to RSE translator	Enum
1	Type mapping	LINQ to RSE translator	Nullable vs NULL ternary logic in SQL
1	Type mapping	RSE to SQL translator	Implicit numeric casts
1	Type cast	LINQ to RSE translator	"is" operator
1	LINQ methods	Both	Concat LINQ method
1	LINQ methods	Both	Union LINQ method
1	LINQ methods	Both	Except LINQ method
1	LINQ methods	Both	Intersect LINQ method
1	LINQ methods	LINQ to RSE translator	GroupBy with aggreagates
1	LINQ methods	LINQ to RSE translator	GroupJoin
1	LINQ methods	LINQ to RSE translator	First, Single, Last, *OrDefault as subquery result (including .Select(x => new {x, x.Customers.First}) )
1	Issue	LINQ to RSE translator	EntitySet.Count property & EntitySet.Contains method
1	Issue	LINQ to RSE translator	EntitySet as IQueryable
1	Issue	LINQ to RSE translator	Correct column order for OrderBy
2	Type mapping	LINQ to RSE translator	Byte array
2	Type cast	LINQ to RSE translator	OfType LINQ method
2	Type cast	LINQ to RSE translator	Cast LINQ method
2	Type cast	LINQ to RSE translator	"as", (T) cast operators for type downcasting
2	Type cast	LINQ to RSE translator	"as" cast operator for type upcasting
2	Type cast	LINQ to RSE translator	(T) cast operator for type upcasting must lead to an error on query translation
2	Optimization	RSE optimizer	SQL APPLY rewriting
2	Optimization	RSE optimizer	Redundant join remover
2	Optimization	LINQ to RSE translator	Join first selected subquery returning * in .Select projections (including .Select(x => new {x, x.Customers}) )
2	Optimization	RSE optimizer	Index access optimization: pi.Filter(...) -> si1.Range(...).Union(si2.Range(...)) .Union(...).Join(pi, ...) .Select(...).Filter(...)
2	Issue	LINQ to RSE translator	Take, Skip after OrderBy
3	LINQ methods	LINQ to RSE translator	Contains, Any, All for parameters of collection (IEnumerable) type

Most part of tasks of priority 1 are in development now, as well as Index access optimization (priority is 2). We're planning to finish with priority 1 queue during the next week. Priority 2 & 3 queue will require 1.5-2 more weeks, but must of these features aren't really necessary now. We're working on all of them mainly to get all our LINQ tests passing.

Btw, most likely it's a very important question: how do we test our LINQ implementation? You may find that now there are ~ 470 different tests for LINQ. Some of them are written by us, but a significant part of them was imported from the following projects:
- IQToolkit (The LINQ IQueryable Toolkit)
- ADO.NET Team's 101 LINQ Samples.

That's why we're pretty sure we'll cover all aspects of LINQ in the end.

Schema evolution

The progress here is not as good as with LINQ - from the point of end users nothing works here yet. But we already implemented the following parts of the framework:
- Generic modelling framework (Xtensive.Modelling) - a set of abstract base classes allowing to describe any models similar to SQL metadata.
- Update actions support for models. Basically, we can track & pass only the changes made to some model, rather then model itself to a remote server.
- Differential comparison support for models. Allows to create hierarchical difference structure taking into account comparison hits (such as node rename), as well as convert it to a set of update actions. This part isn't fully finished yet (some bugs still exist).
- Xtensive.Storage.Indexing.Model - an implementation of indexign storage model based on Xtensive.Modelling
- Converter of SQL DOM model to Indexing model (partially done: some bugs exist)
- Converter of Xtensive.Storage.Model model (Storage model) to Indexing model (in development).

What's left:
- Translation of differences to SQL update actions (for SQL storages)
- Translation of differences to index structure update actions (for indexing storages)
- Integration of all this stuff into Domain.Build pipeline.

Hopefully first version of our schema evolution layer will be available by the end of the next week.

LINQ in DataObjects.Net v4.0: details, issues and news, part 2

2009-03-23T04:01:00.017+05:00

3. Support for common LINQ workarounds for SQL syntax

I really like how LINQ is designed - its syntax is so unified, but still more natural and flexible than SQL. And because of this some SQL features looks quite differently in it - although more naturally. Let's see how LEFT OUTER JOIN may look like:

Version 1:
from p in Products
from nonEmptyC in (from c in Categories where c.CategoryID==p.CategoryID select c).DefaultIfEmpty()
select new {nonEmptyC, p}

The same, but in extension method calls:
Products.SelectMany (
p => Categories.Where (c => c.CategoryID == p.CategoryID).DefaultIfEmpty(),
(p, nonEmptyC) => new { nonEmptyC, p } )

Version 2:
from p in Products
join c in Categories on p.CategoryID equals c.CategoryID into jc
from nonEmptyC in jc.DefaultIfEmpty()
select new {nonEmptyC, p}

The same, but in extension method calls:
Products
.GroupJoin(Categories, p => p.CategoryID, c => c.CategoryID, (p, jc) => new {p, jc} )
.SelectMany ( tmp => tmp.jc.DefaultIfEmpty(), (tmp, nonEmptyC) => new {nonEmptyC, tmp.p} )

So as you see, there are at least two different ways to implement the same LEFT OUTER JOIN in LINQ. And we must recognize both.

4. Optimization of RSE queries for indexing providers

As far as you might know, we translate any query into RSE query first. RSE uses indexes as lowest level abstraction of data sources. But there are two types of indexes:
- Primary indexes - in our case they have the same meaning as table in SQL (because we always have primary keys an any tables).
- Secondary indexes - they allow to locate primary keys by values of other columns from primary indexes they are bound to.

Our LINQ translation layer (as well as any other pat of DO40) "thinks" of primary indexes as of tables they're bound to - if it needs a data from some table, it queries corresponding primary index. Sexondary indexes are ignored at all by LINQ translator. Why? Because LINQ translator's goal is to produce the crudest query plan (RSE query). In particular, this plan:
- Uses only primary indexes
- Uses only index scans. Index seeks (RecordSet.Seek and RecordSet.Range operations) aren't used at all.

Further this plan (RSE query) is either:
- Passed directly to RSE compiler - this happens in underlying execution engine supports query optimization. In fact, this happens for SQL providers.
- Optimized and passed to RSE compiler - this happens for indexing providers.

So we perform query plan optimization. That's what any good SQL server does. And the most interesting quersion here is what exactly our query optimizer can do, and how our query optimizer can be compared with others. Brief answer:
1) For now we're mainly working on index usage related optimizations - e.g. replacement of PrimaryIndex.Filter(...) chain to SecondaryIndex1.Range(...) .Union(SecondaryIndex2.Range(...)) .Union(...) .Join(PrimaryIndex, ...) .Select(...).Filter(...) chain. We hope this will be working in the beginning of April.
2) Further plans include operation reordering, join sequence and aggregate calculation optimizations (earlier I wrote we already maintain some additive aggregates in indexes, so that's how we'll use them).
3) If part 1) will be implemented well, we'll "beat" many embedded database engines right on start - e.g. SQLite (its query optimizer implements this part, but with quite serious limitations). Btw, we're seriously considering implement query plan branching here - this will allow us to gracefully handle queries like "Select ID, Value from Simplest where (ID=@P1) or (@P2=1)" - most of RDBMS (including Microsoft SQL Server) can't do this now. If you don't see anything suspecious in this query, check out the description here.

So we hope our query optimizer will be good enough for embedded databases almost from scratch. It should provide expected optimization results for 95% of relatively simple queries and possibility to write a query translated to any desirable execution plan, if optimizer can't help.

LINQ team short term goals

We must show:
- DO perfectly works with regular databases - in particular, this means we must provide full LINQ support for them
- Embedded databases provided by DO are reliable and fast as well. Initially "fast" means "typical LINQ queries are executed without unnecessary index scans".

That's it.

P.S. This month I feel myself quite devoted to development - last week I worked overnight for few times. That's because I feel the progress, and thus feel myself much more happier ;) In September we switched a part of DO4 team to a new project that was requiring this to be completed in time, and now these guys are back. Moreover, we've hired few new developers after NY, and for now they joined DO4 team as well. So right now there are 7 developers + me working on DO4 - that's much more then 3 developers + me before. This pushes the development forward quite fast.

LINQ in DataObjects.Net v4.0: details and issues

2009-03-10T20:56:00.009+05:00

I think it's right time to give some details on implementation of LINQ in new DO. Let's start from issues:

1. Correlated subqueries. LINQ allows us to write generally anything in from clause (collectionSelector expression in SelectMany) - e.g. a correlated subquery: from a in MainQuery from b in Subquery select new {a, b} (translated by C# to MainQuery.SelectMany(a => Subquery, (a, b) => new { a = a, b = b } )). Note that Subquery is generally any IQueryable, including e.g. AnotherQuery.Where(a.X==...).

Standard SQL doesn't allow to do the same - you can't use correlated subqueries in from clause there. Quite strange, because generally this doesn't significantly affect on anything. But:
- SQL Server 2005 supports this via Cross / Outer apply statement.
- We also suppport this in RSE by ApplyProvider.

It's easy to implement its executable counterpart, but its translation to SQL is rather tricky: we've added this statement to SqlDom, although now we can complile it only for SQL Server 2005. We hope it will be possible to do this without serious limitations for Oracle and PostgreSQL as well, although we aren't going to rely on this: we'll translate correlated subqueries (actually this is possible for the most frequently used part of them) to RSE using JoinProvider or PredicateJoinProvider - in fact, we expect correlated subquery looks like NonCorrelatedSubquery.Where(correlatedExpression) in this case. In other cases we'll use ApplyProvider.

Btw, EF and LinqToSql behave the same (nearly - e.g. there is no RSE behind them ;) ).

2. DateTime and TimeSpan handling. Unfortunately, SQL Server doesn't support interval (or similar) data type, but many others do, so functions dealing with dates are quite different. We did a lot to handle this:
- First of all, we implemented corresponding abstraction layer in SqlDom.
- Secondly, we implemented unified method call translation framework - so-called MemberCompilerProvider - a base class in Xtensive.Core.Linq providing compilers (methods) for specified members and allowing to invoke them fast during the translation process. Now it allows us (and you!) to define compilers operating at any desirable translation layer (in this case, at RSE to SQL translation layer) for generally any invoked members. See sources in Xtensive.Storage.Providers.Sql.Mappings.FunctionMappings namespace (internal!) for many many examples. Here is just one:

[Compiler(typeof(DateTime), "Year", TargetKind.PropertyGet)]
public static SqlExpression DateTimeYear(SqlExpression this_)
{
return SqlFactory.Extract(SqlDateTimePart.Year, this_);
}

... To be continued.

DataObjects.Net v4.0: status update

2009-03-10T15:38:00.010+05:00

Installer: finally this part is almost fully finished. The newest installer (check out the latest nightly build) downloads and installs all you need to either use or even compile new DataObjects.Net. The only feature we should add is test database creation (although this isn't necessary for running samples on IMDB), but this is for some later time.

LINQ support: in the end of February we've started to refactor our implementation of LINQ translator for the third time. First attempt was too crude - mainly for testing. We've finished with it in december. The second one was good, but not simple enough (i.e. too complex to deal with it further). Finally we've come to an approach allowing us to get enough simple unified implementation. This should lead LINQ translation project to completion by the end of March. Obviously, expect query optimizer for index storages - this part will be in development for few more months, but as it was mentioned, absence of this feature for now won't lead to any troubles.

Schema evolution: unfortunately, we're just starting this part. But the good news is that we decided to switch to index-based schema comparison rather then SqlDom-based. This must simplify everything a lot (index model is much simpler then SqlDom database model), and we anyway need this part developed till the launch of our upcoming file system storage provider. But on the other hand, this implies we will need to use a set of workarounds to make schema evolution layer fully compatible with the general SQL database. But for now the implementation speed is more important then internal unification - we still want to deliver this feature till the end of March.

Nightly builds: now they're live for DO4. The file with empty name here is nightly build of DO4 - for now our website improperly recognizes its display name, since we've switched to different naming pattern (to be fixed today).

Other features: almost untouched yet, mainly because of complexities with LINQ translation. It goes much slower then we initially planned, but in comparison to other known implementations we're rather far from reaching the averages - e.g. LLBLGen team spent 6 months on this. And I really don't want to reach this border.

DataObjects.NET v4.0 is updated

2009-02-09T16:39:00.006+05:00

Brief list of new features:
- Partial LINQ support. As you might know, LINQ is work in progress for us, so for now its implementation is really partial. For example, you can't use left loins, SelectMany and so on. But it already works. Further details on this will be published here shortly.
- All the samples are "ported" to LINQ.
- New installer. It is capable of downloading and installing all the dependencies required by DO (for example, PoshSharp) and installs DataObjects.Net Project Template. It can't install source code for now - we have rather limited time on it. But shortly it will.
- Many other changes, mainly in Storage project. AFAIK almost all the bugs reported to us are fixed.

Happy downloading! Hopefully our server won't fall... Ironical laughting :}

DataObjects.NET v4.0: development plan update

2009-02-02T13:03:00.005+05:00

This week: release DO40 with basic LINQ support. We'll support most necessary Queryable methods. Support of ElementAt, Except, Intersect, Union, SequenceEqual, SkipWhile, TakeWhile, OfType, Reverse and correlated subqueries will be either limited or unavailable.

This month (February 2009): almost complete implementation of LINQ, scheme evolution and basic synchronization.

Next month (March 2009): improvements in usability and stability of above features. Quite likely - something else, e.g. full-text indexing and search.

From 2008 to 2009: results and plans

2009-01-26T09:45:00.017+05:00

I'm reader of Joel Spolsky blog. One of the latest posts there exposes the goals of his company for the next year. I instantly understood we should return back to writing such an article once per year - as you might know, we did this before. And it's a good idea to provide a brief overview of what was done in previous year - our activities, results and so on. So let's start.

Done in 2008

We showed DataObjects.Net 4.0 - frankly speaking, alpha. Although its currently available version isn't practically usable (no schema evolution & LINQ), it shows how it will finally look like. Unfortunately we didn't reach a good progress in sales yet - all the income generated by DO now is mainly based on upgrade subscriptions, and frankly speaking, it isn't comparable well to our outsourcing business. So the cost of ~ 2 year pause in its development (i.e. a cost of its complete redesign) is really high. Returning these investments is one of Y2009 goals.

Both MEScontrol.net and custom software solutions departments grew up - most part of our income is generated by these teams now. We're developing projects with MESware, Urals State Techincal University (USTU), Data-Center (for SeverStal) and Izenda.

We launched new X-tensive.com and MEScontrol.ru web sites - so we've got a completely new look. Old x-tensive.com was serving us well for a long time, but its time had come.

We started to promote MEScontrol.net in Russia. This must help to built a new business for us in the next year.

Financially this year was also successful - we've got ~ 120% growth in income.

We've got much better local exposure - I admit rare .NET developer here knows nothing about us now. Partially - because of sessions on .NET User Group we run. I became the leader of Urals .NET User Group as well.

Unfortunately we discarded the idea to bring LiveObjects.Net live, at least for now. LiveObjects.Net is MVC framework and control library for ASP.NET utilizing ExtJS (AJAX). We've been using it internally to build all the projects for USTU. I'll provide more details further.

Goals for 2009

Really polished DataObjects.Net v4.0 will be delivered by the middle of this year. This includes at least:
- LINQ support - that's what we're working on right now
- Schema evolution - the next critical feature we'll deliver
- Synchronization - obviously, it is also one of the most critical ones
- Full-text indexing and search - also an important one
- Transactional file system storage - in fact, our own implementation of RDBMS
- Early preview of distributed storage provider. Its complete implementation must be ready till the end of Y2009. Why so late? We decided to implement all the other features first, and show distributed storage only in some simple form - as a preview. This must be enough to convince most of developers (you) it will really be delivered further, so you can rely on its features. But for now (i.e. in development) you can start using new DO with some other provider instead of it.

First real projects based on v4.0:
- The first one will be developed by our Software Solutions team, but for (or with) some other company - right now we're beginning to explore the opportunities here. A part of this team finishes its current contract with USTU in March; moreover, in March DO 4 will get its final shape, so it's a good moment to seek for the next project. The next post in our blog will describe this more precisely.
- We also hope to show our own service using DO 4, but it's difficult to say if we'll be able to get its showable version till the end of 2009, taking into account that our R&D team will be mainly busy with DO during nearest 6-8 months.

MEScontrol.net implementations on Russian plants \ factories: as it was mentioned, we're going to apply a lot of efforts related to local MEScontrol.net promotion this year based on a background created for this in 2008.

DataObjects.Net v3.X is updated

2009-01-20T22:30:00.002+05:00

It is a minor update - there is nothing else except bugfixes. But one of issues it resolves is important: full-text rank ("{FullTextRank}") in order by clause was ignored at all because of bug in one of recent v3.9 updates, so full-text search results were unordered.

DataObjects.Net 4 project template

2008-12-18T18:45:00.023+05:00

Project template location

Update: now project template is installed by DataObjects.Net installer automatically.

Installation

The installation is simple: just double-click on downloaded "DataObjects.Net_4_Project.vsi" file and follow the installer instructions.

Be sure that DataObjects.Net 4 is installed in the right folder ("C:\Program Files\X-tensive.com\DataObjects.Net 4.0\").
If no, the simplest way to fix it is to copy DataObjects.Net 4 into the specified folder.

Solution for geeks and advanced users

Find "DataObjects.Net 4.0 Project.zip" in Visual Studio 2008 custom project template folder ("%My Documents%\Visual Studio 2008\Templates\ProjectTemplates\Visual C#\").
Unzip contents, open "DataObjects.Net 4.0 Project.vstemplate", set $DataObjectsDotNet4Path$ variable to desired path, save file and pack it back.

Usage

To create new project based on DataObjects.Net 4 framework open Visual Studio, click "New" -> "Project" and select "DataObjects.Net 4 Project" as it is shown on the image:

After clicking "OK" simple C# project will be created. It will contain one persistent class "Person", a set of references to required assemblies and PersistentAttribute will be applied in AssemblyInfo.cs file. Remember to synchronize PersistentAttribute parameters if you are going to rename assembly with your persistent model.

Any comments and suggestions are welcome.

Minor DataObjects.Net 4.0 update

2008-11-25T16:34:00.003+05:00

We've just updated v4.0 installer - the updated version resolves some issues related to compilation of AggregateProvider in RSE.

Other good news: shortly we'll migrate DataObjects.Net code to Google Code. The reason is obvious: really open source project must be hosted on really open source servers ;)

From now DataObjects.Net 4.0 issue tracker is already there.