365bet官网注册开户
引擎使用
📅 2025-07-29 15:50:21
👤 admin
👁️ 5537
❤️ 374
Sqoop
Sqoop 引擎主要依赖 Hadoop 基础环境，如果该节点需要部署 Sqoop 引擎，需要部署 Hadoop 客户端环境，以及下载安装 Sqoop 客户端。
在执行 Sqoop 任务之前，先在该节点使用原生的 Sqoop 执行测试任务，以检测该节点环境是否正常。
Linkis系统参数参数备注wds.linkis.hadoop.site.xml设置sqoop加载hadoop参数文件位置一般不需要单独配置，默认值”core-site.xml;hdfs-site.xml;yarn-site.xml;mapred-site.xml”sqoop.fetch.status.interval设置获取sqoop执行状态的间隔时间一般不需要单独配置，默认值为5s
方式一：直接下载引擎插件包
方式二：单独编译引擎插件（需要有 maven 环境）
# 编译cd ${linkis_code_dir}/linkis-engineconn-plugins/sqoop/mvn clean install# 编译出来的引擎插件包，位于如下目录中${linkis_code_dir}/linkis-engineconn-plugins/sqoop/target/out/
将 2.1 中的引擎包上传到服务器的引擎目录下
${LINKIS_HOME}/lib/linkis-engineconn-plugins
通过重启 linkis-cg-linkismanager 服务刷新引擎
cd ${LINKIS_HOME}/sbinsh linkis-daemon.sh restart cg-linkismanager可以查看数据库中的 linkis_engine_conn_plugin_bml_resources 这张表的 last_update_time 是否为触发刷新的时间。
#登陆到 `linkis` 的数据库 select * from linkis_cg_engine_conn_plugin_bml_resources;`mysql` 导入到 `hive` 库 `linkis_test_ind.test_import_sqoop_1` ,表 `test_import_sqoop_1` 不存在 需要添加参数 `--create-hive-table` sh linkis-cli-sqoop import -D mapreduce.job.queuename=dws \--connect jdbc:mysql://127.0.0.1:3306/casion_test \--username hadoop \--password password@123 \--columns user_id,user_code,user_name,email,status \--fields-terminated-by ',' \--hive-import --create-hive-table \--hive-database casionxia_ind \--hive-table test_import_sqoop_1 \--hive-drop-import-delims \--delete-target-dir \--input-null-non-string '\\N' \--input-null-string '\\N' \--verbose ;`mysql` 导入到 `hive` 库 `linkis_test_ind.test_import_sqoop_1` ,表 `test_import_sqoop_1` 存在 移除参数 `--create-hive-table `sh linkis-cli-sqoop import -D mapreduce.job.queuename=dws \--connect jdbc:mysql://127.0.0.1:9600/testdb \--password password@123 \--columns user_id,user_code,user_name,email,status \--fields-terminated-by ',' \--hive-import \--hive-database linkis_test_ind \--hive-table test_import_sqoop_1 \--hive-overwrite \--hive-drop-import-delims \--delete-target-dir \--input-null-non-string '\\N' \--input-null-string '\\N' \--verbose ;OnceEngineConn 的使用方式是通过 LinkisManagerClient 调用 LinkisManager 的 createEngineConn 的接口，并将代码发给创建的 Sqoop 引擎，然后 Sqoop 引擎就开始执行，此方式可以被其他系统进行调用，比如 Exchangis 。 Client 的使用方式也很简单，首先新建一个 maven 项目，或者在您的项目中引入以下的依赖
org.apache.linkis linkis-computation-client
参数key说明sqoop.modeimport/export/…-Dmapreduce.job.queuenamesqoop.env.mapreduce.job.queuename—connect sqoop.args.connectSpecify JDBC connect string—connection-manager sqoop.args.connection.managerSpecify connection manager class name—connection-param-file sqoop.args.connection.param.fileSpecify connection parameters file—driver sqoop.args.driverManually specify JDBC driver class to use—hadoop-home sqoop.args.hadoop.homeOverride $HADOOP_MAPRED_HOME_ARG—hadoop-mapred-home sqoop.args.hadoop.mapred.homeOverride $HADOOP_MAPRED_HOME_ARG—helpsqoop.args.helpPrint usage instructions-PRead password from console—password sqoop.args.passwordSet authentication password—password-alias sqoop.args.password.aliasCredential provider password alias—password-file sqoop.args.password.fileSet authentication password file path—relaxed-isolationsqoop.args.relaxed.isolationUse read-uncommitted isolation for imports—skip-dist-cachesqoop.args.skip.dist.cacheSkip copying jars to distributed cache—username sqoop.args.usernameSet authentication username—verbosesqoop.args.verbosePrint more information while working
参数key说明—batchsqoop.args.batch Indicates underlying statements to be executed in batch mode—call sqoop.args.callPopulate the table using this stored procedure (one call per row)—clear-staging-tablesqoop.args.clear.staging.tableIndicates that any data in staging table can be deleted—columns sqoop.args.columnsColumns to export to table—directsqoop.args.directUse direct export fast path—export-dir sqoop.args.export.dirHDFS source path for the export-m,—num-mappers sqoop.args.num.mappersUse ‘n’ map tasks to export in parallel—mapreduce-job-name sqoop.args.mapreduce.job.nameSet name for generated mapreduce job—staging-table sqoop.args.staging.tableIntermediate staging table—table sqoop.args.tableTable to populate—update-key sqoop.args.update.keyUpdate records by specified key column—update-mode sqoop.args.update.modeSpecifies how updates are performed when new rows are found with non-matching keys in database—validatesqoop.args.validateValidate the copy using the configured validator—validation-failurehandler sqoop.args.validation.failurehandlerValidate the copy using the configured validator—validation-threshold sqoop.args.validation.threshold Fully qualified class name for ValidationThreshold—validator sqoop.args.validatorFully qualified class name for the Validator
参数key说明—check-column sqoop.args.check.columnSource column to check for incremental change—incremental sqoop.args.incrementalDefine an incremental import of type ‘append’ or ‘lastmodified’—last-value sqoop.args.last.valueLast imported value in the incremental check column
参数key说明—enclosed-by sqoop.args.enclosed.bySets a required field enclosing character—escaped-by sqoop.args.escaped.bySets the escape character—fields-terminated-by sqoop.args.fields.terminated.bySets the field separator character—lines-terminated-by sqoop.args.lines.terminated.bySets the end-of-line character—mysql-delimiterssqoop.args.mysql.delimitersUses MySQL’s default delimiter set: fields: , lines: \n escaped-by: \ optionally-enclosed-by: ‘—optionally-enclosed-by sqoop.args.optionally.enclosed.bySets a field enclosing character
参数key说明—input-enclosed-by sqoop.args.input.enclosed.bySets a required field encloser—input-escaped-by sqoop.args.input.escaped.bySets the input escape character—input-fields-terminated-by sqoop.args.input.fields.terminated.bySets the input field separator—input-lines-terminated-by sqoop.args.input.lines.terminated.bySets the input end-of-line char—input-optionally-enclosed-by sqoop.args.input.optionally.enclosed.bySets a field enclosing character
参数key说明—column-family sqoop.args.column.familySets the target column family for the import—hbase-bulkloadsqoop.args.hbase.bulkloadEnables HBase bulk loading—hbase-create-tablesqoop.args.hbase.create.tableIf specified, create missing HBase tables—hbase-row-key sqoop.args.hbase.row.keySpecifies which input column to use as the row key—hbase-table sqoop.args.hbase.tableImport to in HBase
参数key说明—hcatalog-database sqoop.args.hcatalog.databaseHCatalog database name—hcatalog-home sqoop.args.hcatalog.homeOverride $HCAT_HOME—hcatalog-partition-keys sqoop.args.hcatalog.partition.keysSets the partition keys to use when importing to hive—hcatalog-partition-values sqoop.args.hcatalog.partition.valuesSets the partition values to use when importing to hive—hcatalog-table sqoop.args.hcatalog.tableHCatalog table name—hive-home sqoop.args.hive.homeOverride $HIVE_HOME—hive-partition-key sqoop.args.hive.partition.keySets the partition key to use when importing to hive—hive-partition-value sqoop.args.hive.partition.valueSets the partition value to use when importing to hive—map-column-hive sqoop.args.map.column.hiveOverride mapping for specific column to hive types.HCatalog import specific options:—create-hcatalog-tablesqoop.args.create.hcatalog.tableCreate HCatalog before import—hcatalog-storage-stanza sqoop.args.hcatalog.storage.stanzaHCatalog storage stanza for table creation
参数key说明—accumulo-batch-size sqoop.args.accumulo.batch.sizeBatch size in bytes—accumulo-column-family sqoop.args.accumulo.column.familySets the target column family for the import—accumulo-create-tablesqoop.args.accumulo.create.tableIf specified, create missing Accumulo tables—accumulo-instance sqoop.args.accumulo.instanceAccumulo instance name.—accumulo-max-latency sqoop.args.accumulo.max.latencyMax write latency in milliseconds—accumulo-password sqoop.args.accumulo.passwordAccumulo password.—accumulo-row-key sqoop.args.accumulo.row.keySpecifies which input column to use as the row key—accumulo-table sqoop.args.accumulo.tableImport to in Accumulo—accumulo-user sqoop.args.accumulo.userAccumulo user name.—accumulo-visibility sqoop.args.accumulo.visibilityVisibility token to be applied to all rows imported—accumulo-zookeepers sqoop.args.accumulo.zookeepersComma-separated list of zookeepers (host:port)
参数key说明-conf sqoop.args.confspecify an application configuration file-D sqoop.args.Duse value for given property-fs sqoop.args.fs-jt sqoop.args.jt-files sqoop.args.filesspecify comma separated files to be copied to the map reduce cluster-libjars sqoop.args.libjarsspecify comma separated jar files to include in the classpath.-archives sqoop.args.archivesspecify comma separated archives to be unarchived on the compute machines.
