hadoop - Sqoop import throws java.lang.ClassNotFoundException where class refers to table name

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
I'm using Ubuntu 22.04 and I built a Hadoop server with a sample database in MySql Server which has an orders table.
I'm using Sqoop to generate an import with the following command:
sqoop import --table orders --connect jdbc:mysql://localhost:3306/classicmodels --username hadoop -password 1234 --target-dir /BigData/Sample/Orders/myBirthdayOrders.txt --where "orderDate = '2003-02-17'" --bindir ./libjars
I'm using libjars root directory to save Hadoop generated classes but after run the previous command, I got the following error:
Java.lang.Exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class orders not found
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class orders not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)
    at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:403)
    at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.createDBRecordReader(DataDrivenDBInputFormat.java:270)
    at org.apache.sqoop.mapreduce.db.DBInputFormat.createRecordReader(DBInputFormat.java:266)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:528)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: Class orders not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2686)
But the generated classes is in the libjars directory:
libjars directory content
This is the whole command log:
hadoop@HadoopMaster:~$ sqoop import --table orders --connect jdbc:mysql://localhost:3306/classicmodels --username hadoop -password 1234 --target-dir /BigData/Sample/Orders/myBirthdayOrders.txt --where "orderDate = '2003-02-17'" --outdir ./libjars --bindir ./libjars
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/lib/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
2023-03-12 18:55:18,249 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2023-03-12 18:55:18,335 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
2023-03-12 18:55:18,476 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
2023-03-12 18:55:18,476 INFO tool.CodeGenTool: Beginning code generation
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2023-03-12 18:55:19,283 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `orders` AS t LIMIT 1
2023-03-12 18:55:19,393 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `orders` AS t LIMIT 1
2023-03-12 18:55:19,405 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/hadoop
Note: ./libjars/orders.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2023-03-12 18:55:21,371 INFO orm.CompilationManager: Writing jar file: ./libjars/orders.jar
2023-03-12 18:55:21,385 WARN manager.MySQLManager: It looks like you are importing from mysql.
2023-03-12 18:55:21,386 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
2023-03-12 18:55:21,386 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
2023-03-12 18:55:21,386 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
2023-03-12 18:55:21,390 INFO mapreduce.ImportJobBase: Beginning import of orders
2023-03-12 18:55:21,391 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2023-03-12 18:55:21,518 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
2023-03-12 18:55:22,083 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2023-03-12 18:55:22,199 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2023-03-12 18:55:22,310 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2023-03-12 18:55:22,310 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2023-03-12 18:55:22,640 INFO db.DBInputFormat: Using read commited transaction isolation
2023-03-12 18:55:22,720 INFO mapreduce.JobSubmitter: number of splits:1
2023-03-12 18:55:22,973 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1418369330_0001
2023-03-12 18:55:22,974 INFO mapreduce.JobSubmitter: Executing with tokens: []
2023-03-12 18:55:23,316 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-hadoop/mapred/local/job_local1418369330_0001_e15ca564-f245-408f-925d-501daf15ec63/libjars <- /home/hadoop/libjars/*
2023-03-12 18:55:23,325 INFO mapred.LocalDistributedCacheManager: Localized file:/tmp/hadoop/mapred/staging/hadoop1418369330/.staging/job_local1418369330_0001/libjars as file:/tmp/hadoop-hadoop/mapred/local/job_local1418369330_0001_e15ca564-f245-408f-925d-501daf15ec63/libjars
2023-03-12 18:55:23,476 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2023-03-12 18:55:23,483 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2023-03-12 18:55:23,488 INFO mapreduce.Job: Running job: job_local1418369330_0001
2023-03-12 18:55:23,547 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2023-03-12 18:55:23,547 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2023-03-12 18:55:23,550 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2023-03-12 18:55:23,701 INFO mapred.LocalJobRunner: Waiting for map tasks
2023-03-12 18:55:23,702 INFO mapred.LocalJobRunner: Starting task: attempt_local1418369330_0001_m_000000_0
2023-03-12 18:55:23,772 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2023-03-12 18:55:23,779 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2023-03-12 18:55:23,856 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2023-03-12 18:55:23,899 INFO db.DBInputFormat: Using read commited transaction isolation
2023-03-12 18:55:23,901 INFO mapred.MapTask: Processing split: 1=1 AND 1=1
2023-03-12 18:55:23,935 INFO mapred.LocalJobRunner: map task executor complete.
2023-03-12 18:55:23,988 WARN mapred.LocalJobRunner: job_local1418369330_0001
java.lang.Exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class orders not found
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class orders not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)
    at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:403)
    at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.createDBRecordReader(DataDrivenDBInputFormat.java:270)
    at org.apache.sqoop.mapreduce.db.DBInputFormat.createRecordReader(DBInputFormat.java:266)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:528)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: Class orders not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2686)
    ... 12 more
2023-03-12 18:55:24,547 INFO mapreduce.Job: Job job_local1418369330_0001 running in uber mode : false
2023-03-12 18:55:24,548 INFO mapreduce.Job:  map 0% reduce 0%
2023-03-12 18:55:24,551 INFO mapreduce.Job: Job job_local1418369330_0001 failed with state FAILED due to: NA
2023-03-12 18:55:24,556 INFO mapreduce.Job: Counters: 0
2023-03-12 18:55:24,560 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
2023-03-12 18:55:24,561 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 2.4241 seconds (0 bytes/sec)
2023-03-12 18:55:24,561 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2023-03-12 18:55:24,561 INFO mapreduce.ImportJobBase: Retrieved 0 records.
2023-03-12 18:55:24,562 ERROR tool.ImportTool: Import failed: Import job failed
                Sqoop is a retired Apache Project. Are you required to use it? Spark is more commonly used to read JDBC tables into HDFS nowadays
– OneCricketeer
                Mar 15 at 21:23
                And not really clear how you created this libjars folder, but you should try using Maven/Gradle instead. Classes probably cannot be imported from the default package (show your orders class, please)
– OneCricketeer
                Mar 15 at 21:25
        Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.