So we've had long discussions about Linq, XLinq, and particularly DLinq.
It's probably a good exercise to forget about databases, ORMs, and XML files, and spur the masses into thinking where else they think Linq would be useful. Maybe someone here will take up one of the suggestions as a project...
FooLinq
Ravi Chand
NHiberLINQ
Translates LINQ queries to Hibernate Query Language (HQL) used in NHibernate (www.nhibernate.org).
In mi opinion, DLINQ currently is a good bottom up development style tool, where you start with an existing database and then generate "classes" that map exactly each table definition with Sqlmetal. But DLINQ is not very well suited for top down development style, where you create first a rich fine grained domain model and then you generate the database from the mapping attributes or XML, using CreateDatabase(). NHibernate does very well in the later approach, supporting all 3 strategies for mapping inheritance, all types of associations including appart from one-to-many, one-to-one and many-to-many, fine grained composition for dependant value objects, polymorphism in queries, custom user types, etc., so more than using DLINQ as an ORM, I would prefer to use LINQ to generate type safe, intellisense enabled queries that translate themself to HQL when executed, so I can use all the power of NHibernate for rich fine grained domain models.
Maybe DLINQ will support this top down development style in the future, but meanwhile I'm with NHibernate, so NHiberLINQ would be a good project.
Mark Atkinson
That's cool. What about Linq to any website Say Microsoft came up with a layer that glues a website database to a representation that Linq can understand
mfatih
What do you think
trying
I was *just* pondering about using Linq against my own XML parser, which for various reasons, I wrote from scratch to support the immense amount of data I need to work with in my science research To introduce, here's its main characteristics:
1. Parses XML streams (sometimes GB-sized), outputs tables in "columns." As a consequence of working with streams, we avoid buffering rows. We output them as soon as they come in. The performance of the parser currently seems to be bottlenecked by I/O only (as opposed to the parser code or memory management).
2. User specifies columns as xml tags, which may be specified without having to specify full path from root tag.
3. Optionally, generate statistics on columns (avg, stdev, min, max, etc). Here buffering of rows do come into play, since non-statistics columns act as keys.
4. Simple computed values. "x" and "y" and "log(x*y)" can be three columns, computed from <x>data</x> and <y>data</y> in the stream.
5. Filtering: don't output rows where computed values eval to 0.
6. XML inclusion: XML files can include other XML files.
7. Table compression: instead of the bloat from <row><col>data</col></row>, the "schema" for the table is expressed, and the entire table (must be complete, i.e. non-sparse) is represented as a single sequence of CSV data. XMLParse maintains the illusion of <row><col>data</col></row>. For example, in my work, I have 8000 XML files (included from a master XML file), each with a combination of tables, each 1 megabyte large: A LOT of data, even with this "table compression."
I implemented this with it being a command line app, but I also use it as a C++ class called XMLParse. Another class, XMLReporter, uses XMLParse to further arrange the data. I'm using XMLReporter in a utility program (using C++/CLI and C#) that utilizes Wolfram Research's Mathematica to generate plots, either to a Winform window or to various file formats (e.g. animated GIFs). Here I'm combining the flexibility of the XMLParse tool to arbitrarily specify data to be plotted: XML tag for x-axis, y-axis, 2nd y-axis, z-axis, error bars from stdev, 2-D, 3-D, slider animations, Excel-like series behavior, etc.
It is an interesting exercise to think about wrapping this stuff with LINQ: much of the concepts are the same. I can't visualize it much right now, just throwing this out. One caveat is that the XML parser is written in C++. Another caveat that this all must work with streaming.
Brian
David Gwynn-
Hi,
Databases are useful in cases where managing concurrency is an issue and the amount of data is large.
Quite a few applications out there have no more than a couple of megabytes of data (if even that). These apps can easily hold all their data in memory as objects. LINQ can be every useful in quering this in memory object store.
Ronald Cormier
jbacigal
Hmm...:
1) FooLinq to query the number of Foo-fighters in space at any one time.
2) Create ControlLinq to query and set electrical control systems. Maybe HouseLinq, or CarLinq.
3) Naturally, linq over WS-Management would be cool to query over and inventory multiple systems as one virtual db.
4) SnifferLinq to query IP traffic over some period of time.
5) MusicLinq to query the notes in an mp3 or wav.
dan_goita msft
Cool. My best friend is in astronomy. I actually editorialized in my last post that Microsoft needs to start getting serious about science scenarios, but felt it was too off-topic so I deleted it. The .NET stuff is amazing, but the incremental benefit for the science community would be far greater than for the existing business scenarios because 1. science produces more data, and 2. the state of the art in processing data is generally abysmal (people either don't care or are not aware of new programming approaches to managing large amounts of data).
I've use my XML for 1. genetic algorithm perf tracking over generations relative to a bunch of possible parameters. 2. (currently) protein folding analysis.
Back to your questions. I do use C++/CLI (for which I'm eternally grateful). My XML parser is in C++ for Unix compat. I wrote some interesting abstraction classes for wrapping Mathematica in C#. I tied these things together in C++/CLI to do the XML plotter.
The XML parser, when used as a class, currently uses a callback. The callback in C++/CLI isn't too hard, but it's not obvious either. I may try doing a yield return and employing the enumerator pattern. It would be fun to watch, because the xml parser relatively complex (i've only used yield return on simple collections that can be iterated with a for loop.)
No schema. It just walks the XML, keeping track of certain things as it goes. I have a tutorial doc that I started a long time ago, but is incomplete, that describes how the tool is used.
It's not XPath, but I'm sure there are things in common. But for various reasons, mostly subjective, I went with my own query interface (implying scanning implementation) that seemed more natural for me. I run the queries on the command line a lot, so terseness was a goal. Implementing a psuedo and definitely partial XPath language/scanner didn't appeal to me either. People I work with don't really see the value of XML anyway, and perputate the usual stdin/stdout/a-bunch-of-perl-and-awk-in the middle style of working with data. So I'd never have any of my peers wondering why I'm not using XPaths. :)
As an aside, I now use XML to represent command line parameters with one of the biggest benefits being to support iterators (lists, loops, etc). The XML gets embedded in the C++ executables and is parsed there, comprising what might arguably be called a "poor mans" refection layer. The apps to call copies of themselves on a Linux/Mac cluster, using XML as both inputs and outputs, etc etc. The end-to-end scenario begins with me specifying an experiment with multiple dimensions, and ends with me getting a single XML document, perhaps days later. Analyzing that experimental data is where I'm still open to new ideas, including LINQ.
Brian
Jim Thompkins
But this is an XML ecosystem for me, especially from the fact that the data generated is from apps running on a Linux/Apple cluster.
My intention wasn't to ask others what a better storage mechanism would be. It was merely to let Keith know that i'm also thinking of Linq in a different context. Although he meant something other than XML, i think my case is interesting in that it is my own XML parser.
Bran
good metric
My background's actually physics and astronomy, so stuff like this can be intrinsically interesting to me. ;)
Re C++: Have you tried porting this to C++/CLI
Re Streams: I've wrapped a incoming bytestream using IEnumerable<T>, where T is the deserialized type before. It wasn't too difficult, and worked well for the application in question.
Re your schema: Is there an independant schema description You could try runtime type generation and an interface to manage the deserialization into a CLR type.
Re LINQ over your schema: At first glance, it doesn't sound like it'd be difficult to achieve basic LINQ over it. It does sound, though, like this system doesn't support XQuery- or XPath-like scanning, which only limits the types of queries you can use. Of course, if you have no need to use them it's no loss.
daaboots
How about replacing WIQL (Work Item Query Language) in VSTS with LINQ
Another possibility would be to replace VSEFL (Visual Studio Event Filter Language) with LINQ.
Both of these are based off SQL, so they would seem to be naturals for LINQ.
-Larry
Sanchi
concisely and declaratively with Linq