exo-ecm-dms

Big files uploading and memory increasing

Details

  • Type: Task Task
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.1
  • Fix Version/s: 1.1
  • Component/s: None
  • Labels:
  • Similar issues:
    ECM-1620 Increasing version is wrong in special case
  • Description:
    Hide

    Need to implement logic for the big files uploading. JCR supports big files uploading without "Out of memory" error but ECM doesn't. Could u pls modify file UIUploadForm.java, line 90 (and related files):

    contentNode.setProperty(JCR_DATA, new ByteArrayInputStream(content));

    we should use another stream than ByteArrayInputStream (e.g. InputStream).
    Because of array of bytes transformations we have "Out of memory" error.
    If u'll implement this logic we could upload big files (750M and greater) such as video films etc. without "Out of memory" error.

    Show
    Need to implement logic for the big files uploading. JCR supports big files uploading without "Out of memory" error but ECM doesn't. Could u pls modify file UIUploadForm.java, line 90 (and related files): contentNode.setProperty(JCR_DATA, new ByteArrayInputStream(content)); we should use another stream than ByteArrayInputStream (e.g. InputStream). Because of array of bytes transformations we have "Out of memory" error. If u'll implement this logic we could upload big files (750M and greater) such as video films etc. without "Out of memory" error.

Activity

Hide
Peter Nedonosko added a comment - 24/Oct/06 9:47 AM

The problem is not in "Out of memory" error. It's a consequense of reading BLOB data in memory (ByteArrayInputStream).

eXo JCR has internal mechanism of use BLOBs in Property.setValue(InputStream) and Node.setProperty(String,InputStream), ValueFactory.createValue(InputStream value) methods.

So, you have to give InputStream to JCR only, no more. If it's a HTTP form upload give a InputStream obtained from app.server direct without any 'caching' etc. JCR has own cache/swap logic for InputStreams

Show
Peter Nedonosko added a comment - 24/Oct/06 9:47 AM The problem is not in "Out of memory" error. It's a consequense of reading BLOB data in memory (ByteArrayInputStream). eXo JCR has internal mechanism of use BLOBs in Property.setValue(InputStream) and Node.setProperty(String,InputStream), ValueFactory.createValue(InputStream value) methods. So, you have to give InputStream to JCR only, no more. If it's a HTTP form upload give a InputStream obtained from app.server direct without any 'caching' etc. JCR has own cache/swap logic for InputStreams
Hide
Gennady Azarenkov added a comment - 24/Oct/06 10:59 AM

It seems to the problem may come from org.exoplatform.faces.core.component.UIUploadInput
FileItem item = (FileItem) iter.next();
......
.......
byte[] buf = item.get();

in a case of huge file uploading it may cause OutOfMemoryException...
may be better try to use item.getInputStream() instead...

We do need this issue fixed in 1.1

Show
Gennady Azarenkov added a comment - 24/Oct/06 10:59 AM It seems to the problem may come from org.exoplatform.faces.core.component.UIUploadInput FileItem item = (FileItem) iter.next(); ...... ....... byte[] buf = item.get(); in a case of huge file uploading it may cause OutOfMemoryException... may be better try to use item.getInputStream() instead... We do need this issue fixed in 1.1
Hide
Nguyen Quang Hung added a comment - 01/Nov/06 8:33 AM

I changed to use Node.setProperty(String, InputStream) as

byte[] content = input.getContent();
ValueFactoryImpl valueFactory = (ValueFactoryImpl) uiExplorer.getSession().getValueFactory() ;
Value contentValue = valueFactory.createValue(content) ;
.......
contentNode.setProperty(JCR_DATA, contentValue.getStream());

Still get problem...

java.lang.OutOfMemoryError: Java heap space
[ERROR] JDBCIOChannel - Can't write value data of property with id a26802030a00001f012df636928f64fd,
error:java.sql.SQLException: out of memory <java.sql.SQLException: out of memory>java.sql.SQLExcept
ion: out of memory
at org.hsqldb.jdbc.jdbcUtil.throwError(Unknown Source)
at org.hsqldb.jdbc.jdbcPreparedStatement.executeUpdate(Unknown Source)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedState
ment.java:101)
at org.exoplatform.services.jcr.impl.storage.jdbc.JDBCIOHelper.writeValueData(JDBCIOHelper.j
ava:423)
at org.exoplatform.services.jcr.impl.storage.jdbc.JDBCIOHelper.write(JDBCIOHelper.java:386)
at org.exoplatform.services.jcr.impl.storage.jdbc.JDBCStorageConnection.add(JDBCStorageConne
ction.java:281)
at org.exoplatform.services.jcr.impl.dataflow.persistent.WorkspaceDataManager.doAdd(Workspac
eDataManager.java:354)
at org.exoplatform.services.jcr.impl.dataflow.persistent.WorkspaceDataManager.save(Workspace
DataManager.java:95)
at org.exoplatform.services.jcr.impl.dataflow.persistent.VersionableWorkspaceDataManager.sav
e(VersionableWorkspaceDataManager.java:170)
at org.exoplatform.services.jcr.impl.dataflow.session.TransactionableDataManager.save(Transa
ctionableDataManager.java:254)
at org.exoplatform.services.jcr.impl.dataflow.session.TransactionableDataManager.saveItem(Tr
ansactionableDataManager.java:292)
at org.exoplatform.services.jcr.impl.dataflow.session.TransactionableDataManager.saveItem(Tr
ansactionableDataManager.java:303)
at org.exoplatform.services.jcr.impl.core.SessionDataManager.commit(SessionDataManager.java:
410)
at org.exoplatform.services.jcr.impl.core.ItemImpl.save(ItemImpl.java:239)
at org.exoplatform.portlets.content.jcr.component.UIUploadForm$SaveActionListener.execute(UI
UploadForm.java:95)
at org.exoplatform.faces.core.event.ExoActionListener.processAction(ExoActionListener.java:6
7)

Show
Nguyen Quang Hung added a comment - 01/Nov/06 8:33 AM I changed to use Node.setProperty(String, InputStream) as byte[] content = input.getContent(); ValueFactoryImpl valueFactory = (ValueFactoryImpl) uiExplorer.getSession().getValueFactory() ; Value contentValue = valueFactory.createValue(content) ; ....... contentNode.setProperty(JCR_DATA, contentValue.getStream()); Still get problem... java.lang.OutOfMemoryError: Java heap space [ERROR] JDBCIOChannel - Can't write value data of property with id a26802030a00001f012df636928f64fd, error:java.sql.SQLException: out of memory <java.sql.SQLException: out of memory>java.sql.SQLExcept ion: out of memory at org.hsqldb.jdbc.jdbcUtil.throwError(Unknown Source) at org.hsqldb.jdbc.jdbcPreparedStatement.executeUpdate(Unknown Source) at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedState ment.java:101) at org.exoplatform.services.jcr.impl.storage.jdbc.JDBCIOHelper.writeValueData(JDBCIOHelper.j ava:423) at org.exoplatform.services.jcr.impl.storage.jdbc.JDBCIOHelper.write(JDBCIOHelper.java:386) at org.exoplatform.services.jcr.impl.storage.jdbc.JDBCStorageConnection.add(JDBCStorageConne ction.java:281) at org.exoplatform.services.jcr.impl.dataflow.persistent.WorkspaceDataManager.doAdd(Workspac eDataManager.java:354) at org.exoplatform.services.jcr.impl.dataflow.persistent.WorkspaceDataManager.save(Workspace DataManager.java:95) at org.exoplatform.services.jcr.impl.dataflow.persistent.VersionableWorkspaceDataManager.sav e(VersionableWorkspaceDataManager.java:170) at org.exoplatform.services.jcr.impl.dataflow.session.TransactionableDataManager.save(Transa ctionableDataManager.java:254) at org.exoplatform.services.jcr.impl.dataflow.session.TransactionableDataManager.saveItem(Tr ansactionableDataManager.java:292) at org.exoplatform.services.jcr.impl.dataflow.session.TransactionableDataManager.saveItem(Tr ansactionableDataManager.java:303) at org.exoplatform.services.jcr.impl.core.SessionDataManager.commit(SessionDataManager.java: 410) at org.exoplatform.services.jcr.impl.core.ItemImpl.save(ItemImpl.java:239) at org.exoplatform.portlets.content.jcr.component.UIUploadForm$SaveActionListener.execute(UI UploadForm.java:95) at org.exoplatform.faces.core.event.ExoActionListener.processAction(ExoActionListener.java:6 7)
Hide
Vitaliy Obmanyuk added a comment - 16/Nov/06 3:37 PM

I've been commited new class placed in
D:\java\projects\v2.x\exo-portal\jsf\core-components\src\java\org\exoplatform\faces\core\component\UIUploadInputStream.java
and modified UIUploadForm, lines 47 and 73 (UIUploadInput replaced with UIUploadInputStream)
Now we may upload big files (like 800Mb) using ECM (make sure exo-jcr-cofig.xml are currently configured for the <value-storages> etc. tags).

Show
Vitaliy Obmanyuk added a comment - 16/Nov/06 3:37 PM I've been commited new class placed in D:\java\projects\v2.x\exo-portal\jsf\core-components\src\java\org\exoplatform\faces\core\component\UIUploadInputStream.java and modified UIUploadForm, lines 47 and 73 (UIUploadInput replaced with UIUploadInputStream) Now we may upload big files (like 800Mb) using ECM (make sure exo-jcr-cofig.xml are currently configured for the <value-storages> etc. tags).

People

Dates

  • Created:
    24/Oct/06 9:24 AM
    Updated:
    16/Nov/06 3:37 PM
    Resolved:
    16/Nov/06 3:37 PM
    Date of First Response:
    24/Oct/06 9:47 AM