Paul Kiel's Data Integration Blog

Fourth data standards consortia

2009-04-16T20:42:00.004-04:00

I've been talking directly with my 4th data standards consortia (all listed below). It seems I am getting a reputation for this. The fourth is a new organization that is dedicated to world food traceability. Recent news stories about problems have illustrated the need to find a solution. News reports of Peanut butter in Minnesota and tomatoes in the southwest show an outbreak of salmonella. But finding the source of these outbreaks has proven difficult. The folks at World Food Trace are working to create an effective solution that is very cost effective.

And in fact, my work with OAGi has begun to branch out because of its role as a platform on which other standards can build. So a convergence is occurring, which is connecting me to groups like CIDX, UN/CEFACT, STAR, AIAG, ISA, WBF, PDES (STEP) and others.

XBRL, HL7 and the rise of domain data models

2009-02-23T14:21:00.004-05:00

Been reading several items relating to the new proposal to require the use of XBRL for financial reporting. Indeed, Kurt Cagle has an extensive post, saying "this should be interesting." He suggests that this development should have a number of effects, including an increased need for Xml skills. Even Alexander Falk of XmlSpy (actually the company name is Altova but their flagship app is xmlspy) has commented on his company's product native support for both XBRL and HL7.

While I haven't worked intimately with XBRL, I have looked at their library from a design / best practice perspective as it relates to the other Xml Consortia for whom I have consulted. Its ontological nature is interesting and I think has something to say about that kind of data model.
The reason I group HL7 in there is because of the new administration's effort to push for electronic records in health care. This doesn't name HL7 by name, but one can't help but connect the two. Now having worked with a number of Xml Consortia, from human resources to supply chain management to media to research administration, I have always been amazed at the contrast between how technologically advanced the product is (medical care) and how *under*-automated the infrastructure is.

So I'll end the post by saying that I hope the trend is for the better. There is so much really good work being done in these domain Consortia and I'd love to see them have a broader impact. Yes, they each have their quirks (and not all of them are worth implementing), but the promise of domain data models has always been good. And there are many of us out there who are working to make these Consortia adopt common design patterns. Now we need to roll up our shirt sleeves and make them more of a reality.

Schema Lightener 2.0 released

2009-02-03T11:13:00.007-05:00

Announcing a new release of the “Xml Schema Lightener” utility. It has been completely updated and is now version 2.0.

What it does
----------------
It takes an Xml Schema and creates a subset based on user input. Some refer to this with terms such as a “subset Schema view”, “pruned Schema”, “LiteBOD”, or “profiled Schema”.

Why it does this
----------------
Consider these use cases:

You have an Xml Schema and want to implement a subset (a selected portion) of the data points. Or you want to communicate the selected data points to an audience.
You have an internal data model in Schema and want to communicate to a trading partner only the data points necessary for a particular integration.
You are implementing an Xml Schema acquired from somewhere else (i.e. from a trading partner, or from a standards consortium such as OAGi, HR-XML, etc.) and want to implement only the data points relevant to your business context.

In all these cases, it would be useful to create a Schema that shows these points and can be used for validation. The Xml Schema Lightener allows you to create the contextual Schema that each of these require. It does it using simple XSLT and Xml. And a GUI is included to make it easy to use.

It has been heavily tested, includes a free flattener, and also has a simple GUI too.

Cost effective
----------------
This utility is presented with its full functionality in a very inexpensive package. The $29 (USD) price does not cover the full cost of development and is simply a way to offset a portion of its creation and keep it going.

For more information
----------------
See http://www.xmlhelpline.com/tools/ for more information and FAQ.

the joys of air travel

2009-02-03T08:56:00.004-05:00

Cost of checked bag: $25

Cost of "emergency" exit seat: $30

In flight meal: $5

Bottle of water: $2

Airline customer service: priceless

Solved! internal_power_error BSOD on hibernate

2008-12-21T18:18:00.009-05:00

Strange title for the posting I know, but if you are having the same problem you'll recognize it. I've had this intermittent problem with hibernation and done a lot of research around it. It all began when HP replaced my motherboard because of a recall issue. It came back in good working order. However, whenever I tried to hibernate (on vista), it caused a BSOD (blue screen of death) and automatically rebooted. Not 100% of the time but the vast majority of the time.

I want hibernate because the laptop is in my bedroom and sleep mode allows a flashing blue light to continue indefinitely. At night, I don't want this flashing, so hibernate is perfect for my needs.

Did a bunch of research on this problem, which lots of other folks have had. Many said there was a process that somehow caused the machine to come out of hibernate immediately. Others said it was a BIOS issue. Still others said completely deactivate hibernation and then reactivate. The more adventurous of you can read more here, here, here, here, here, and here (even though I have 32 bit).

Here are some of the things I've tried:
Tried powercfg.exe /hibernate off and then back on. No effect. On power settings, completely disabled hibernate on all modes (power button, lid close, etc), then reactivated. No fix. I went into device manager and made sure none of the devices were "allowed to wake computer". No fix. Also checked task scheduler and removed any task that I thought could be responsible for waking the machine at the wrong time. Again no fix.

Someone suggested a corrupt hibernation file. Trashed it and had it create another. Not a fix. My BIOS is completely updated, so I didn't suspect that. And I didn't want to regress to an earlier version. (Also someone said this was fixed in SP1, but I already had that installed.)

So my solution came in this form. First, I disabled automatic restart. This allowed me to see the BSOD message before it rebooted. The error was "internal_power_error", which is of course horribly unclear. Here is how to disable auto restart from a help file:
1. Click on Start and then Control Panel.
2. Click on the System and Maintenance link.
3. Click on the System link.
4. In the task pane on the left, click the Advanced system settings link.
5. Locate the Startup and Recovery area and click on the Settings button.
6. In the Startup and Recovery window, locate and uncheck the check box next to Automatically restart.
7. Click OK in the Startup and Recovery window.
8. Click OK in the System Properties window.
9. You can now close the System window.
10. From now on, when a problem causes a BSOD or another major error that halts the system, the PC will not automatically reboot. Rebooting manually will be necessary.

Second, on my NVIDIA nForce Networking Controller (the culprit), I had previously made sure it was not allowed to wake computer. (Unchecked "Allow this device to wake computer" in power management tab.) The problem was that my system was trying to turn the device off to save power. In device manager, I "unchecked" the "Allow the computer to turn off this device to save power" box (see pic below, which has it checked as I found it originally). This seems to have fixed the problem. Not sure exactly why this was a problem, but I am happy nevertheless. I've been able to hibernate for a few days now with no errors. Hope this helps some of you out there with the same problem.

SchemaLightener gets a user interface

2008-12-10T16:26:00.002-05:00

The Xml SchemaLightener now has a user interface. So making a thinner view of an Xml Schema is now as simple as point and click.
Screenshot (click to enlarge - image is a bit fuzzy):

The SchemaLightener can be integrated into other tools and so does not *require* the use of the UI. But the UI provides a nice easy way to perform this function. Need a quick view of the schema for a group meeting? No problem. Just point and click, selecting an Xml Schema to lighten, an Xml instance that is representative of what you want, and then executing the transformation.

This UI is built with Java, so it works cross platform. It has been tested on windows vista, xp, and linux (ubuntu). And as I said before, has been tested using many Consortia standard schemas.

Plus, the Flattener, which was recently added, can also be invoked using the UI.

SchemaLightener gets a Flattener

2008-11-19T11:03:00.007-05:00

I've added a new feature to the Xml SchemaLightener, namely that of a Flattener. First, here are some definitions:

Lighten: to remove elements and types from a schema, while the remaining elements and types are still consistent with the original schema. To "prune", or create a subset of the schema. To create a "view" of the schema, focusing on certain components.

Flatten: to merge all schema dependencies connected via xsd:include into a single schema file. To create a "stand alone" version of a set of schemas. All components are consolidated into one file without any editing or changes. Does not pertain to dependencies connected via xsd:import (because this will not result in a valid schema).

I've often needed a flattening tool. Schemas are usually developed modularly to maximize reuse. This has tremendous benefits in building a data model. However, come deployment time, tools sometimes have a problem managing a web of xsd:include files. So it is generally recommended to create a runtime version of a schema which consolidates includes into a "stand alone" one. (Of course it may not literally be stand alone because of xsd:import files, but it isn't possible to merge these according to the spec.)

The difference between xsd:include and xsd:import is in namespaces. Files connected via xsd:include have the same default namespace. This is what allows them to be consolidated into a single schema. Files connected via xsd:import have different default namespaces.

So I've thrown in an XSLT which will create a stand alone or flattened schema into the SchemaLightener tool. Importantly, even though imports cannot be merged, it does automatically go to each import and flatten its includes (if any) and so the result is a fully valid set of schemas that may have imports but no includes.

In short, the absolute minimum number of schema files possible.

Schema Lightener undergoes heavy testing

2008-10-22T18:37:00.015-04:00

The Xml Schema Lightener is coming along quite nicely. I've been gathering as many test cases as I can to put a significantly updated version through trials. It is designed to lighten a schema (some call it "prune" or "filter" or "creating a view of") based on an xml instance. Here are the test cases I've run:

127 test cases ... HR-XML (Human Resources Xml) v. 2_5
58 test cases ... HR-XML (Human Resources Xml) v. 3.0 (draft)
494 test cases ... OAGi (Open Applications Group) v. 9.2
141 test cases ... STAR (Standards for Technology in Automotive Retail) v. 5.1.4
56 test cases ... CIDX (Chemical Industry Data eXchange) v. 4.0
96 test cases ... ACORD (Insurance Data Standards) v. 2.9
289 test cases ... OTA (Open Travel Alliance) v. 2003-05
41 test cases ... TWIST (Transaction Workflow Innovation Standards Team) v 3.1

In total, that is 1,302 test cases. All of them are lightened correctly. All of them result in valid xml schemas. All of the original test xml validates against the resulting schema. 100%!

This process has really helped me increase the quality of this tool. It is a great resource for removing components that are never used from an xml schema.

HR-XML integration summit next week

2008-10-06T15:44:00.004-04:00

Next week is the second annual Global Partnering and Integration Summit. It is designed to bring together the HR community and solve data integration issues. Last year's show was very successful. (You can still register for this one.)
I'll be involved in the pre-conference workshop entitled:

Making Data Standards Work:
A Detailed Look at Pre-hire to Post-hire HR Systems Integration

It's been a pleasure working with ADP and HES Corp on this session. Hope you can join us in Chicago.

article: Extend enumerated lists in XML schema

2008-09-23T16:33:00.004-04:00

Hi folks,

In my effort to empower Xml Schema designers and implementers, I've added another item of interest. I've written an article for IBM's DeveloperWorks on the topic of extensible enumerated lists.

Extend enumerated lists in XML schema

The topic of extending enumerated lists is a perennial request I get from clients. The fact that the schema spec does not allow for this extensibility has caused many to develop a work around. The article outlines solutions I've seen.

It is a complimentary piece to my article on Xml.com some time ago on developing a usable profile of xml schema.

Profiling Xml Schema

This paper was well received and I even got a couple clients as a direct result of the article. So I feel like my effort to empower Xml Schema designers is proving useful for folks.

My current work helping schema implementers is involved in moving the Xml Schema Lightener into a more mature level. Again, I got some great early interest and am working hard on improvements.

OAG and STEP convergence

2008-07-31T17:56:00.003-04:00

Just back from a convergence meeting between OAGi and STEP (Standard for the Exchange of Product Model Data). Lots of players including OMG PLM services, OASIS PLCS TC, Eurostep, PDES inc, etc. This is one of many convergence efforts building upon OAGIS as a platform. It is great to see so many standards coming together and doing the right thing. Sometimes the standards world can get distracted with egos and agendas and competition (to put it nicely). But it seems like convergence is the name of the game now and the rush to build upon OAGIS as a platform is becoming a stampede. Lots of work to do, but that is a good problem to have.

Some great upcoming news on OAGIS as a platform, so stay tuned!

STAR meeting

2008-06-24T09:51:00.006-04:00

I am off today to participate in the STAR meeting in california. They are the consortium for standards in automotive retail. STAR has had some really great success in advocating and enabling data integration standards. They build their material on top of OAGIS, thus leveraging an existing data model. This is similar to what HR-XML is doing as well as others. (Look ma, standards convergence!)
I have more to say about OAGIS as a platform later as there is some interesting news. But for the time being, I'm off to STAR.

Microsoft gets behind UML

2008-06-11T09:28:00.011-04:00

Could be real exciting news. I saw this posting from Joe McKenrick. Microsoft backing UML as part of its SOA (Oslo Modeler) offerings would be a real boon to the UML world. Templates that bring UML to Visio are nice, but getting backing in a bigger way is enticing. Hopefully they are not simply trying to create buzz. A few other links here and here.

Schema Lightener updated, version 1.1.1

2008-05-28T14:33:00.005-04:00

I’ve finally gotten a moment to update the Schema Lightener (now version 1.1.1). Several of you have been requesting this and I am glad to make some progress on it.

Most notably, I’ve fixed the “stack overflow error” that sometimes occurred as a result of how namespaces in data (not tags) is handled by XSLT processors. The problem was when two element names (the value of xsd:element/@name and not the tag itself) were the same but with different namespace prefixes. XSLT can handle the namespace prefix in tags very well, but in the content of data, there are some caveats. And because we are parsing an xml schema, the data is in the content.

Long story short, the Schema Lightener is updated and no longer throws this error. I’ve tested this with OAGIS 9.2 stand alone schemas and got some really good results. 100% validity and significant reduction in the number of components when applied using the sample xml instances that come with it.

For OAGIS folks, when this is combined with the BOD Flattener, it can take a big cumbersome schema set and pair down as well as flatten into something much more manageable.

I would like to know what you want to see next:

1) a user interface of some kind (i.e. browse to the xsd to lighten)
2) ability to take multiple xml instances as input
3) improve performance
4) other ….

Send me an email to request a free copy or let me know what you want next.

off to Helsinki

2008-04-13T11:02:00.001-04:00

I'm off to Helsinki Finland, doing some OAGIS training and consulting work there. My first time to that city, so I am excited.

using Ruby with windows

2008-03-29T10:37:00.001-04:00

I've downloaded Instant Rails (Ruby on Rails), which is a development environment for Ruby in a windows setting. So far it has worked really well. Nice not to have to fuss with dual boot issues. And it comes with PHP and MySql in a nice all-in-one package.

Found this little gotcha in that if you are in windows, chances are good something already has port 80 taken. Edit your config to use another port. I have IIS already, so I had to change ports. Here is a blog post on this topic.

you asked, I delivered: Schema Lightener updated

2008-03-19T21:59:00.002-04:00

Recently posted about the Schema Lightener tool. Some of you responded with a request to better support Xml Namespace prefixes. Now, you can use prefixes in either the Xml instance or the schema. This new version still uses xslt 1.0 technology. Send me an email and I'll send you a copy. As I've said its completely free.

Also, I've added in a command line XSLT processor, a batch file, and a couple sample schemas to help you get started.

Am also working on a version with xslt2.0 that will manage all your includes and imports with one invocation.

Schema Lightener, a free tool to simplify xml schemas

2008-03-05T15:17:00.006-05:00

I just released the free Xml Schema Lightener in order to address the issue described as "LiteBODs", subset Schemas, or constrained Schemas. In short it is a method to take a Schema and create a "Lite" one with all the undesired elements, attributes, and data types removed. Same data model but with fewer components.

I've gotten this request from clients in the data integration space, who either need to communicate a subset to their others (internal or external); or from clients who are trying to generate code from the Schema and don't want lots of unnecessary classes created for elements they don't implement.

Also got this request from Consortium members (i.e. members of HR-XML and OAG), where they want to take a base open standard Schema and constrain it down to an "as implemented" one that can be used for validation. It's been tested with OAGIS version 9.1 and HR-XML version 2.5.

There is an FAQ and real world statistics on the amoung of reduction in the number of components. Have a look and let me know your thoughts.

the mighty spreadsheet

2008-02-25T17:04:00.004-05:00

Working in technology for a while, I find it interesting that with all the tech available to folks, the simplest approach is so often used.

Take data integration. In the process of creating requirements to be modeled (whether in Xml Schema or UML), mapping data to internal processes, or mapping data between systems, folks often gravitate to the simple spreadsheet as the primary work horse.

I've done this with many clients, using variations of similar spreadsheets to do mappings and requirements gathering. I've seen standards folks use this, in HR-XML, OAGi, UBL, and the CCTS folks. They all invariably use the spreadsheet over much more sophisticated tools.

Hail the humble spreadsheet!

great Ruby podcast online

2008-02-22T07:55:00.004-05:00

Great podcast on Ruby with author Russ Olsen, who wrote Design Patterns in Ruby. Have not yet read the book, but will surely do so as I get more experience. The podcast is free and I found it to be very good.

programming in Ruby

2008-02-15T17:02:00.006-05:00

I've been hearing so much about Ruby, I finally went out and bought a book. It's called Learning Ruby by Michael Fitzgerald. I found the book pretty good at laying out the jist of Ruby. The most interesting thing about it is that everything is an object. Even numbers are classes! This takes object orientation to its logical conclusion.
I also like the "dynamic typing" wherein the system determines the data type of a variable. While I wonder if this leads to casting problems, it is nice to have someone else take care of the details.
Will post on a Ruby podcast next.

Xml arrives at age 10

2008-02-15T16:32:00.003-05:00

Hard to believe but Xml is now 10 years old. It has been an amazing decade of development. Those of us with a love (illness?) for this technology are happy to see it become so implemented. I remember when it was first published in 1998, I was trying to implement it with a text editor and the "Panorama" plugin viewer. I had no way to validate at the time. Then within a year, I was implementing it again with the new Xml support in Internet Explorer 5. Tools are so much better now.

Tim Bray has just written a great piece called "Xml People" about Xml's development in the early days and the people who were there.

For those who want to understand the original spec, Tim also wrote this great annotated version of the spec. I've referred to this numerous times over the years.

Relatedly, the spec is currently at Xml 1.0 fourth edition. There is a proposal to make the unsuccessful version 1.1 into an Xml 1.0 fifth edition. But as I've blogged, it has its detractors.

Happy birthday Xml!

Impact of XML on Data Modeling

2008-02-11T15:13:00.000-05:00

An interesting thread on the xmlschema-dev list about the impact of xml on data modeling.

Scott Tsao starts off talking about how to define different kinds of data models. The Conceptual, Logical, and Physical. He says that Xml Schema is the logical model and the binding to a database would be the physical one.

Essam Mansour says: "Conceptual: the model at this level is platform-independent and also could be logically modeled using any modeling technique, ERD, OO model, or XML Schema. Logical: is also platform-independent but it is based on specific modeling technique. Physical: is platform dependent."
This strikes me as pretty close on. Xml Schema can do any of these models, it is the semantic concepts that are different.

Anthony B. Coates says what I think is very common and have come across many times. He says Xml Schema is the physical and UML is the conceptual. This is in fact how they each tend to be used. But I have also found strong support for Xml Schema at the conceptual level. Anthony goes on to say that the conceptual (UML) is the business representation and the physical (XSD) is the technicians' view. This again is what I have come across with my clients. But it is not rigid. The line between business and technical is often blurry, and reality creeps in. But the broad separation of these if often seen as an "ideal".

Michael Kay, who is everywhere and one of the smartest folks I've read, says the concepts are separate. A conceptual model gives you abstraction that can benefit data modeling. So while the concepts are separate, there is an underlying standardized data model.

This is where already standardized data models such as OAGIS and HR-XML come in to play. They can give you a heads up on creating a new model (whether in UML or in XSD) or they can be considered an open source data sharing model for B2B transactions.

With some support for the conceptual=UML and physical=XSD, the discusssion then goes on to ask "can xsd also model the conceptual?" See Scott Tsao.

I totally agree with Scott in that it "can" play that role. And I've seen it done that way. The bottom line is to use the modeling tool that is most usable and comfortable to you. If it is UML, then use it. If XSD, then use it. I've seen it done both ways and I don't think there are any musts here.

The big problem with UML and XSD is translating between them. But I'll blog on that topic separately.

OAGIS 9.2 released today

2008-02-11T12:20:00.001-05:00

Just announced that OAGIS version 9.2. is out today. Go grab a copy while supplies last...

Is XML 1.0 (5th ed) backwardly compatible?

2008-02-09T16:36:00.000-05:00

Been reading alot about the Xml 1.0 fifth edition, now a "Proposed Recommendation". The reasoning goes like this: there are no Xml 1.1 docs because parser writers believe no one will use it and no one will use it because parser writers don't support it. So why not make Xml 1.1 into an erratum version of Xml and call it 5th edition of 1.0? That seems to be what has been done.

Someone recently wrote about a problem of backward compatibility. The issue is around Unicode. Originally Xml was explicitly tied to Unicode 2. But Unicode has not stood still, moving on to version 4 and beyond. So what if someone wants to use Unicode 4 in a xml setting?

The theory is a relaxation of constraints. Xml currently says Unicode 2 and anything not explicitly allowed is prohibited. The Xml 1.0 5th edition says anything not explicitly prohibited is allowed. So following the logic, the 5th edition is a superset of the previous editions in terms of allowed characters.

Michael Rys says that 1.1 becoming a recommendation was a day for mourning. I assume he would say the same for 1.0 5th edition.

Norm Walsh likes it alot. "The fifth edition does not change the status of any existing XML 1.0 document with respect to well-formedness or validity. Nor does it introduce any of the backwards-incompatible changes introduced in XML 1.1."

So Norm's comment makes me think it fully backwardly compatible (the 5th edition that is).

John Cowan describes the characters are not allowed in 5th edition.

David Carlisle says the change is a good one, but it should be called explicitly a version and not passed off as an erratum.

Mark Nottingham describes the problem that most seem concerned about, namely "implementation Z (of, say, the 3rd edition) coming across a 5th edition document and blowing up".
This is a logical concern. But even this concern is still backwardly compatible in my reading as long as the Unicode standard is expanding in super sets and not altering existing material. This is what I understand to be the case.

So I still haven't found out what makes Xml 1.0 5th edition not backwardly compatible. For sure it would lead to things not being forwardly compatible. But that is a well understood issue in software development and doesn't prevent forward progress.