Store assets in DB instead FileSystem

Topics: Developer Forum
Feb 26, 2009 at 4:25 PM
Hello,

I have created a new HttpHandler to store the assets that I upload to the CMS in a database instead of the file system. But I have found the FileManager from the N2.Edit project needs to be modified and a new table, n2File created in the database.

I believe that by integrating the handler in the N2.Core, and making the File Manager in N2.Edit use a generic HttpHandlerBase interface would be an improvement. We could then choose whether to use a database, filesystem, cloud platform etc depending on a value in the config file.

The reason we have used a database in our projects is so we don't have to replicate the files when we use a load balancer, and for permission issues on the File System.

I can share the code with you as I feel it will be an enhancement to the N2CMS solution.

BR,
Pedro
Feb 26, 2009 at 9:43 PM
Keeping UI stuff in the db sounds very interesting -- i've tried experimenting with a similar subject -- a custom Virtual Path Provider without much luck.. Do you think it's feasible to store N2's data in the cloud ? It seems that current schema is very compatible with the SimpleDB and the likes. Given that N2 is capable of dynamically loading assemblies, it might be possible to build a solutions that are completely stored in the cloud: 1) assets, 2) assemblies and 3) "data", in a present sense..
Feb 27, 2009 at 2:20 PM
My proposal to integrate it into N2 CMS is:

In N2.Edit Project, the file \Edit\FileManagement\Default.aspx.cs should use an interface and not access to the file system directly. The interface should have the methods:

IFileManager{
  SiteMapProvider()
  UploadFile()
  CreateFolder()
  DeleteFile() & DeleteDirectory() ?
  Delete()
  DeleteValidation()
}

The method SiteMapProvider shoud return Web.FileSiteMapProvider() if the File System is being used or SqlSiteMapProvider if the assets are stored in the database.

The classes in the next  post.
Feb 27, 2009 at 2:24 PM
Edited Feb 28, 2009 at 4:15 PM
The source code is attached in this ticket: http://n2.codeplex.com/WorkItem/View.aspx?WorkItemId=21431
Coordinator
Feb 27, 2009 at 9:59 PM
Hi, I looked at the code and it looks very interesting. I didn't find any IFileManager. Were you planning on contributing anything similar or should I put it on the list?
Feb 28, 2009 at 4:13 PM
Hi, the IFileManager is not yet created, I am planing about contributing it but I wanted to show first the state of the contribution in order to get some feedback and keep integrating it in the core. Should I do that in the svn version of N2?

Thanks
Pedro
Coordinator
Feb 28, 2009 at 9:16 PM
Great. I actually have some feedback. The site map provider api has been a thorn in my side ever since I used it. It's not really meant for file operations. A viable alternative is the virtual path provider API but as Taras says, it's a bit cumbersome. I'd like to suggest a simpler API that N2 could work against. I committed it to here but it's just a suggestion so take it for what it's worth.
Feb 28, 2009 at 9:42 PM
Thanks for your feedback, the Interface looks good. On the other hand the implementation that I posted to store the assets in the database is using linq, which needs .NET 3.5 maybe could be a good idea to change it to use the same data base access used in N2.Core, to be consitent with the rest of the CMS.

Mar 2, 2009 at 3:19 PM
Hi Cristian, I've been trying to integrate the DatabasePathFileSystem class but I have several doubts.

1. All the factories I've seen in the code are resolved statically in the files *.castle.config. What if I want to resolve it in the code depending of a config value? from web.config for example...

2. For the persistence, where should the extra database be created? running a single sql file and from the install process in the future for example.

3. Should I have to create a N2.Persistence.NH.FilePersister to handle the file storage in the DB?

Thank you in advance for your support and I hope to learn how N2 is structured to be more efficient in future contributions.
Pedro
Coordinator
Mar 2, 2009 at 5:06 PM
Okay, I'm assuming the DBFileSystem is a plug-in that replaces the default functionality.

So you'd need to create a class that implements the IFileSystem interface. The class would probably retrieve the files from the db.

Then you can replace the default implementation in web.config:

<n2><engine><components><add service="N2.Edit:FileSystem.IFileSystem, N2" implementation="DBFileSystem, DBFileSystem"/>...

Lastly you'll need the module to provide the files at the appropriate path's.
Mar 2, 2009 at 9:52 PM
Ok, good assumption, thank you for your help.

On the other hand, to implement DBFileSystem is ok to use linq? Or should be better to use NHibernate to be consistent with the rest of the core? If so, is there any convention to follow? can you tell me about any class that uses NHibernate in a "similar" way?

Thanks again for your support.
Coordinator
Mar 2, 2009 at 10:10 PM
This is to give you an idea of what you might want to use:
  • N2/Persistence/IRepository<K,E>, this is the interface mostly used to load and store entities (e.g. Context.Current.Resolve<IRepository<int,ContentFile>>())
  • Use <n2><database><mappings><add name="My.Resource.hbm.xml, My" to add additional mappings
These might be useful to look at:
  • N2/Persistence/NH/ConfigurationBuilder, creates the session factory (reference only)
  • N2/Mappings/Default.hbm.xml for the entity schema, this is enhanched by types found at runtime (reference only)
NHibernate is very competent ORM (if not the most competent around). I can certanly reccommend you look into it for your own benefit. Just remember that there are more more modern ways to use it, so look around.

Then again. You may build this separately and use entity framework if that's more confortable to you. I'd rather see this based on EF than not at all =)

Mar 4, 2009 at 11:49 AM
Thanks Cristian for your support.

I am implementing the interface using LINQ as entity manager but I am having some problems. As I implement IFileSystem in a separate project inside my VS2008 solution called DataAccess, I include N2.dll as reference. Then in web.config I add:

<components>
  <add service="N2.Edit.FileSystem.IFileSystem, N2" implementation="N2.Edit.FileSystem.DatabaseFileSystem, N2" />
</components>

But when N2.Engine call RegisterConfiguredComponents() the implementation is not yet compiled, and is null and I get an exception, and if I try to add DataAccess as reference to N2 (something that shouldn't be done) I can't because I would get a circular dependency.

Is there any solution to this problem??

Thanks
Pedro


Mar 4, 2009 at 5:32 PM
I have posted some code to the ticket http://n2.codeplex.com/WorkItem/View.aspx?WorkItemId=21431

I've been implementing the IFileSystem interface using Linq but not yet finished. I don't know if I will have enough time to finish it, so maybe I'll go back to the workaround I did the last time.

I attach the code I've done. What I think that should be done is instead creating a handler for the files stored in the database would be a better solution to implement a VirtualPathProvider, when the files are requested via GetFile() check the prefix (/Upload) then call IFileSystem.GetFile() if it's true, otherwise call base.GetFile(). Then the reference from the browser to the assets stored in the database would be a real path, and not a path to a handler.

Thank you for any feedback.
Coordinator
Mar 4, 2009 at 9:34 PM
Virtual path providers sounds like a good idea. Did you solve the configuration issue?
Mar 4, 2009 at 10:47 PM
No, I havent´t solved the configuration issue. What I´ve done is to add the N2.Edit.FileSystem.DatabaseFileSystem class to the N2 core project, and add the reference to the Linq library and the DatabaseModel. But I believe that I should be able to implement the IFileSystem interface from a project just including the N2.dll library without having to modify the core source code. Any ideas about how this can be done?

Thx!
Coordinator
Mar 5, 2009 at 4:06 PM
You could create a class library from visual studio and add a reference to N2.dll
Mar 5, 2009 at 4:09 PM
But it would create the circular dependency. To implement the IFileSystem I need to add the reference to N2.dll in my class library in VS. And N2 need a reference to the implementation of IFileSystem to load in the load process. That is the problem that I was having...

Does it make sense?
Thx
Coordinator
Mar 5, 2009 at 6:47 PM
It makes sense, but you might have missed out that I added the interface to N2.dll. If you get the code from trunk you'll find it. If you use that interface you won't need the any circular dependencies.
Mar 6, 2009 at 8:40 AM
Yeah yeah, I've been using the trunk version of N2, and the project compiles fine. But in runtime in the file N2.Engine.RegisterConfiguredComponents() when it tries to load the implementation of IFileSystem, as the implementation is not in N2.dll, or not yet loaded or I don't really know the reason, it throws an exception because the implementation is NULL.

Thx!
Coordinator
Mar 6, 2009 at 9:52 PM
Can you post the configuration you're using? The implementation should have the format "Namespace.Classname, Assemblyname". There is an implementation in the demo site complete with configuration that could help.