Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I need to consume a rather large amounts of data from a daily CSV file. The CSV contains around 120K records. This is slowing to a crawl when using hibernate. Basically, it seems hibernate is doing a SELECT before every single INSERT (or UPDATE) when using saveOrUpdate(); for every instance being persisted with saveOrUpdate(), a SELECT is issued before the actual INSERT or a UPDATE. I can understand why it's doing this, but its terribly inefficient for doing bulk processing, and I'm looking for alternatives

I'm confident that the performance issue lies with the way I'm using hibernate for this, since I got another version working with native SQL (that parses the CSV in the excat same manner) and its literally running circles around this new version)

So, to the actual question, does a hibernate alternative to mysqls "INSERT ... ON DUPLICATE" syntax exist?

Or, if i choose to do native SQL for this, can I do native SQL within a hibernate transaction? Meaning, will it support commit/rollbacks?

There are many possible bottlenecks in to bulk operations. The best approach depends heavily on what your data looks like. Have a look at the Hibernate Manual section on batch processing.

At a minimum, make sure you are using the following pattern (copied from the manual):

Session session = sessionFactory.openSession();
Transaction tx = session.beginTransaction();
for ( int i=0; i<100000; i++ ) {
Customer customer = new Customer(.....);
session.save(customer);
    if ( i % 20 == 0 ) { //20, same as the JDBC batch size
        //flush a batch of inserts and release memory:
        session.flush();
        session.clear();
tx.commit();
session.close();
If you are mapping a flat file to a very complex object graph you may have to get more creative, but the basic principal is that you have to find a balance between pushing good sized chunks of data to the database with each flush/commit and avoiding exploding the size of the session level cache.
Lastly, if you don't need Hibernate to handle any collections or cascading for your data to be correctly inserted, consider using a StatelessSession.
                I am flushing an clearing my session, I dont have memory issues with the code. I have issues with the extra select! :P I read through the manual, I cant find anything. The data is super simple, no cascading needed. I just need to get rid of the, for this task, redundant select that gets called 120K times :P
– JustDanyul
                Sep 8, 2011 at 15:25
                @JustDanyul what is the approximate percentage of new entities in this operation (i.e. what percentage of the selects are actually unnecessary)? Are you using versioning?
– jcwayne
                Sep 8, 2011 at 17:18
                the actual percentage will variate from day to day. However, none of the selects should really be necessary. Most databases today (even "toy" ones like SQLite) provides functionality which will let you automatically update a record if the data already exists. (without having to poll it first, to find out if it exists :))
– JustDanyul
                Sep 8, 2011 at 18:38
                You're correct that the functionality exists in virtually any DB you'd be likely to use with Hibernate. But, as best as I can tell Hibernate makes using it impossible. If you don't want to bypass Hibernate and execute the SQL directly via JDBC, your only option is to find other ways to speed your overall import process (e.g. concurrency, tuning of connection pool settings, disable 2nd-level cache, etc.).
– jcwayne
                Sep 8, 2011 at 19:53
Session session = sessionFactory.openSession();
Transaction tx = session.beginTransaction();
ScrollableResults employeeCursor = session.createQuery("FROM EMPLOYEE")
                                   .scroll();
int count = 0;
while ( employeeCursor.next() ) {
   Employee employee = (Employee) employeeCursor.get(0);
   employee.updateEmployee();
   seession.update(employee); 
   if ( ++count % 50 == 0 ) {
      session.flush();
      session.clear();
tx.commit();
session.close();
But for insert i would go for jcwayne answer
High-throughput data export
If you only want to import data without doing any processing or transformation, then a tool like PostgreSQL COPY is the fastest way o import data.
Batch processing
However, if you need to do the transformation, data aggregation, correlation/merging between existing data and the incoming one, then you need application-level batch processing.
In this case, you want to flush-clear-commit regularly:
int entityCount = 50;
int batchSize = 25;
EntityManager entityManager = entityManagerFactory()
    .createEntityManager();
EntityTransaction entityTransaction = entityManager
    .getTransaction();
try {
    entityTransaction.begin();
    for (int i = 0; i < entityCount; i++) {
        if (i > 0 && i % batchSize == 0) {
            entityTransaction.commit();
            entityTransaction.begin();
            entityManager.clear();
        Post post = new Post(
            String.format("Post %d", i + 1)
        entityManager.persist(post);
    entityTransaction.commit();
} catch (RuntimeException e) {
    if (entityTransaction.isActive()) {
        entityTransaction.rollback();
    throw e;
} finally {
    entityManager.close();
Also, make sure you enable JDBC batching as well using the following configuration properties:
<property
    name="hibernate.jdbc.batch_size"
    value="25"
<property
    name="hibernate.order_inserts"  
    value="true"
<property
    name="hibernate.order_updates"  
    value="true"
Bulk processing
Bulk processing is suitable when all rows match pre-defined filtering criteria, so you can use a single UPDATE to change all records.
However, using bulk updates that modify millions of records can increase the size of the redo log or end up taking lots of locks on database systems that still use 2PL (Two-Phase Locking), like SQL Server.
So, while the bulk update is the most efficient way to change many records, you have to pay attention to how many records are to be changed to avoid a long-running transaction.
Also, you can combine bulk update with optimistic locking so that other OLTP transactions won't lose the update done by the bulk processing process.
If you use sequence or native generator Hibernate will use a select to get the id:
<id name="id" column="ID">
    <generator class="native" />
You should use hilo or seqHiLo generator:
<id name="id" type="long" column="id">  
    <generator class="seqhilo">
        <param name="sequence">SEQ_NAME</param>
        <param name="max_lo">100</param>
    </generator>
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.