Issue Details (XML | Word | Printable)

Key: ECM-170
Type: Task Task
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Nguyen Quang Hung
Reporter: Vitaliy Obmanyuk
Votes: 0
Watchers: 1
Operations

Add/Edit UI Mockup to this issue
If you were logged in you would be able to see more operations.
exo-ecm-dms

Big files uploading and memory increasing

Created: 24/Oct/06 09:24 AM   Updated: 16/Nov/06 03:37 PM
Component/s: None
Affects Version/s: 1.1
Fix Version/s: 1.1

Time Tracking:
Not Specified

Resolution Date: 16/Nov/06 03:37 PM
Date of First Response: 24/Oct/06 09:47 AM


 Description  « Hide
Need to implement logic for the big files uploading. JCR supports big files uploading without "Out of memory" error but ECM doesn't. Could u pls modify file UIUploadForm.java, line 90 (and related files):

contentNode.setProperty(JCR_DATA, new ByteArrayInputStream(content));

we should use another stream than ByteArrayInputStream (e.g. InputStream).
Because of array of bytes transformations we have "Out of memory" error.
If u'll implement this logic we could upload big files (750M and greater) such as video films etc. without "Out of memory" error.

 All   Comments   Work Log   Change History   Subversion Commits   FishEye      Sort Order: Ascending order - Click to sort in descending order
Peter Nedonosko added a comment - 24/Oct/06 09:47 AM
The problem is not in "Out of memory" error. It's a consequense of reading BLOB data in memory (ByteArrayInputStream).

eXo JCR has internal mechanism of use BLOBs in Property.setValue(InputStream) and Node.setProperty(String,InputStream), ValueFactory.createValue(InputStream value) methods.

So, you have to give InputStream to JCR only, no more. If it's a HTTP form upload give a InputStream obtained from app.server direct without any 'caching' etc. JCR has own cache/swap logic for InputStreams


Gennady Azarenkov added a comment - 24/Oct/06 10:59 AM
It seems to the problem may come from org.exoplatform.faces.core.component.UIUploadInput
      FileItem item = (FileItem) iter.next();
      ......
      .......
        byte[] buf = item.get();

in a case of huge file uploading it may cause OutOfMemoryException...
may be better try to use item.getInputStream() instead...

We do need this issue fixed in 1.1

Nguyen Quang Hung added a comment - 01/Nov/06 08:33 AM
I changed to use Node.setProperty(String, InputStream) as

byte[] content = input.getContent();
      ValueFactoryImpl valueFactory = (ValueFactoryImpl) uiExplorer.getSession().getValueFactory() ;
      Value contentValue = valueFactory.createValue(content) ;
.......
contentNode.setProperty(JCR_DATA, contentValue.getStream());

Still get problem...

java.lang.OutOfMemoryError: Java heap space
[ERROR] JDBCIOChannel - Can't write value data of property with id a26802030a00001f012df636928f64fd,
 error:java.sql.SQLException: out of memory <java.sql.SQLException: out of memory>java.sql.SQLExcept
ion: out of memory
        at org.hsqldb.jdbc.jdbcUtil.throwError(Unknown Source)
        at org.hsqldb.jdbc.jdbcPreparedStatement.executeUpdate(Unknown Source)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedState
ment.java:101)
        at org.exoplatform.services.jcr.impl.storage.jdbc.JDBCIOHelper.writeValueData(JDBCIOHelper.j
ava:423)
        at org.exoplatform.services.jcr.impl.storage.jdbc.JDBCIOHelper.write(JDBCIOHelper.java:386)
        at org.exoplatform.services.jcr.impl.storage.jdbc.JDBCStorageConnection.add(JDBCStorageConne
ction.java:281)
        at org.exoplatform.services.jcr.impl.dataflow.persistent.WorkspaceDataManager.doAdd(Workspac
eDataManager.java:354)
        at org.exoplatform.services.jcr.impl.dataflow.persistent.WorkspaceDataManager.save(Workspace
DataManager.java:95)
        at org.exoplatform.services.jcr.impl.dataflow.persistent.VersionableWorkspaceDataManager.sav
e(VersionableWorkspaceDataManager.java:170)
        at org.exoplatform.services.jcr.impl.dataflow.session.TransactionableDataManager.save(Transa
ctionableDataManager.java:254)
        at org.exoplatform.services.jcr.impl.dataflow.session.TransactionableDataManager.saveItem(Tr
ansactionableDataManager.java:292)
        at org.exoplatform.services.jcr.impl.dataflow.session.TransactionableDataManager.saveItem(Tr
ansactionableDataManager.java:303)
        at org.exoplatform.services.jcr.impl.core.SessionDataManager.commit(SessionDataManager.java:
410)
        at org.exoplatform.services.jcr.impl.core.ItemImpl.save(ItemImpl.java:239)
        at org.exoplatform.portlets.content.jcr.component.UIUploadForm$SaveActionListener.execute(UI
UploadForm.java:95)
        at org.exoplatform.faces.core.event.ExoActionListener.processAction(ExoActionListener.java:6
7)

Vitaliy Obmanyuk added a comment - 16/Nov/06 03:37 PM
I've been commited new class placed in
D:\java\projects\v2.x\exo-portal\jsf\core-components\src\java\org\exoplatform\faces\core\component\UIUploadInputStream.java
and modified UIUploadForm, lines 47 and 73 (UIUploadInput replaced with UIUploadInputStream)
Now we may upload big files (like 800Mb) using ECM (make sure exo-jcr-cofig.xml are currently configured for the <value-storages> etc. tags).