Migrating from Cloudera to Open-Source Hadoop: A Technical Guide
Moving a Cloudera cluster to open-source Hadoop is mostly a management plane problem. The components that run the workloads (HDFS, YARN, Hive, Spark, HBase, Kafka, and the rest) are Apache projects in both cases. What is specific to Cloudera is the layer around them: Cloudera Manager renders every configuration file, starts every daemon, holds the Kerberos identities and TLS material, schedules replication, and exposes the API that operations automation depends on.
This guide describes, at the level of directories, configuration keys, and identities, what changes when that layer is replaced, and how Bifrost handles each part in an in-place Classic migration. The target is the Ilum Bigtop distribution, managed by Apache Ambari (Ilum build), operated directly or through Ilum Cluster Ops. The hosts, the data, and the identities of the cluster stay the same.
What Cloudera Manager Does on a Cluster
| Responsibility | On Cloudera CDP and CDH | After the migration |
|---|---|---|
| Software distribution | Parcels, unpacked under /opt/cloudera/parcels/ and activated by symlink | RPM packages of the Ilum Bigtop distribution, installed per version under /usr/bigtop/ |
| Process supervision | The Cloudera Manager agent with its own supervisord launches every role | The Ambari agent, a systemd service, starts each daemon as its service user and restarts it automatically |
| Configuration rendering | Every launch renders configuration files and keytabs into a generated process directory | Static configuration in /etc/<component>/conf, written by Ambari from the desired configuration |
| Kerberos | Cloudera Manager can create principals and generate keytabs | The existing keytabs are placed under /etc/security/keytabs/; no KDC administrator account is used |
| TLS | Auto-TLS with the Cloudera Manager CA, or corporate PKI | The existing keystores and truststores are reused under /etc/security/serverKeys/ |
| Monitoring and alerts | Service Monitor, Host Monitor, and the alert publisher | Ambari alerts, Prometheus, Grafana, and Loki |
| Replication | Replication Manager | Replication schedules in Apache Ambari (Ilum build) |
| Queue management | YARN Queue Manager | YARN Queue Manager view in Apache Ambari (Ilum build) |
| واجهة برمجة التطبيقات | Cloudera Manager REST API | Ambari REST API, with an optional Cloudera Manager API compatibility layer |
| Console access | Cloudera Manager users and LDAP login | Ambari LDAP login with group-to-role mapping, or Ilum Cluster Ops |
What Happens to Cloudera Manager
- Stopping. In phase 4, the services are stopped through the Cloudera Manager API. The
cloudera-scm-agent,cloudera-scm-supervisordوcloudera-scm-serverunits are then stopped, disabled, and masked on every host. Their previous state is recorded for rollback, and the run stops if any installed unit is left unmasked. - Packages and database. The Cloudera Manager packages and the Cloudera Manager database are not modified, so that a rollback can restart Cloudera Manager against its own database.
- Process directories. The generated process directories under
/var/run/cloudera-scm-agent/process/live on tmpfs and disappear at the next stop or reboot. Phase 3 archives them as a root-only rollback and evidence asset. Every configuration value that points into them is re-pointed to a stable path. - Parcels. Phase 5 removes only the activation symlinks. The parcel content is kept as a rollback asset until finalize.
- Leftovers. A parcel daemon or a process still supervised by Cloudera Manager stops the takeover. A write fence keeps Ambari from writing into the
conf.cloudera.*directories, andvalidate-postreports every Cloudera Manager unit that is not masked.
Package and Directory Layout
| Item | Cloudera CDP | Ilum Bigtop distribution |
|---|---|---|
| Component installation | /opt/cloudera/parcels/CDH/lib/<component> | /usr/bigtop/<version>/usr/lib/<component> |
| Active version | Parcel activation symlink | /usr/bigtop/current/<component>, set with distro-select |
| Daemon configuration | /var/run/cloudera-scm-agent/process/<id>-<service>-<ROLE>/ | /etc/<component>/conf |
| Client configuration | /etc/<component>/conf, an alternatives link to conf.cloudera.<service> | /etc/<component>/conf, an alternatives link to the configuration Ambari writes |
| Keytabs | Rendered into the process directory at each launch | /etc/security/keytabs/ |
| TLS material | Auto-TLS agent certificate directory or corporate paths | /etc/security/serverKeys/ |
| Process supervision | cloudera-scm-agent و cloudera-scm-supervisord | ambari-agent |
Packages of several versions can be installed side by side; distro-select switches the active version for every component at once.
Configuration Translation
Reading the Effective Configuration
Discovery reads the full configuration view of every service, role, role configuration group, and host through the Cloudera Manager API, together with the client configuration bundles, the HDFS nameservices, the replication peers and schedules, and the external accounts. Over SSH, it also captures the files the daemons actually run with: the gateway configuration in /etc/hadoop/conf, hive-site.xml from the HiveServer2 process directory, and capacity-scheduler.xml from the ResourceManager process directory.
When the Cloudera Manager field and the rendered file disagree, the rendered file wins: the Cloudera Manager field describes the intent, the rendered file describes what the daemon really ran with.
Translating Properties
Each property passes through the same ordered rules:
- Properties whose names indicate a secret are replaced by a vault reference.
- Cloudera-internal settings (
com.cloudera.*, Navigator settings, health thresholds, and similar) are filtered out and recorded in the review report. - A property that has no open-source equivalent is dropped when it holds the Cloudera default, and placed in the review queue when it was set by the cluster owner.
- Every other property is routed to an Ambari configuration type by its name. A property without a route is placed in the review queue.
| Property prefix | Ambari configuration type |
|---|---|
dfs. | hdfs-site |
yarn.scheduler.capacity. | capacity-scheduler |
yarn.app.mapreduce. | mapred-site |
hbase. | hbase-site |
spark. | spark-defaults |
tez. | tez-site |
atlas. | application-properties |
Cloudera Manager settings that have no direct property name are translated to the properties they control:
| Cloudera Manager setting | نتيجة |
|---|---|
hdfs_hadoop_ssl_enabled set to true | dfs.http.policy اضبط على HTTPS_ONLY |
yarn_service_lce_always set to true | yarn.nodemanager.container-executor.class set to the LinuxContainerExecutor فصل |
dfs_encrypt_data_transfer_algorithm with an AES cipher suite | Written to dfs.encrypt.data.transfer.cipher.suites |
hiveserver2_webui_bind_wildcard set to true | hive.server2.webui.host اضبط على 0.0.0.0 |
hbase_cell_acl_authorization | Carried, inverted, into hbase.security.access.early_out |
Ports that Cloudera Manager keeps at the role configuration group level are carried as well, for example the HBase Thrift port 9090 and the Spark history server ports 18088 and 18089. After the cutover, every port a source role listened on is checked on the same host.
Safety Valves
Advanced configuration snippets (safety valves) are expanded into individual properties of the configuration type they target:
<!-- HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml -->
<property>
<اسم>dfs.namenode.handler.count</اسم>
<قيمة>200</قيمة>
<final>صحيح</final>
</property>
The snippet above is carried as dfs.namenode.handler.count with the value 200 في hdfs-site, and the final flag is kept.
- Environment snippets (
*_env_safety_valve) are parsed askey=valuelines. Dedicated handlers carry the Spark Python interpreter, the Hue settings, and the KMS JCE settings. - Role-level snippets for the NameNode, DataNode, JournalNode, HBase Master, and RegionServer are carried cluster-wide, or as a configuration group when another layer sets the same key differently.
- A snippet that cannot be parsed unambiguously stops the plan; it is never read as empty.
Role Configuration Groups
Role configuration groups are not copied one to one. Their values are folded into the cluster-wide Ambari configuration types. An Ambari configuration group, named bifrost-<service>-<host>, is created only where the same role runs with different values on different hosts and the difference survives the cutover. Values that exist only because of Cloudera Manager, such as process directory paths and Cloudera Manager macros, never become configuration groups. Plan validation reports every value that was folded.
Guards on Carried Values
- A carried value that still points into the Cloudera installation (
/opt/cloudera,PARCELS_ROOT,/var/run/cloudera، أوconf.clouderadirectory) is refused before the cutover. - After the takeover,
validate-postchecks every Ambari configuration type for references to the old installation tree. - A carried key can be dropped only with a per-key acknowledgement that records who dropped it, why, and when.
Data and Metadata
Data never moves. Before the package swap, the data directories of the NameNode, DataNodes, JournalNodes, Kafka brokers, Solr, Oozie, ZooKeeper, and the KMS are recorded by inode and modification time. Any change after the swap stops the migration and triggers rollback, and so does a Cloudera package that owns a data directory.
| Component | What happens |
|---|---|
| HDFS | The on-disk layout version is compared with the target software. On the version-aligned target they are equal and no NameNode metadata is touched. When they differ, the active NameNode runs the layout upgrade and the standby bootstraps from it. |
| Hive Metastore | The same metastore database is re-pointed. schematool -validate is a gate before the metastore starts. On CDH 6 sources, the schema is upgraded to the Hive 3 schema. |
| HBase | The balancer is turned off, tables are flushed, and HBase is shut down cleanly before the swap. The data stays on HDFS, and the balancer is turned back on after the takeover. The Phoenix indexing WAL codec is carried, and the Phoenix library is installed wherever it is configured. |
| ZooKeeper | The data directory, with every application znode, is kept and never reinitialized. A snapshot is taken for rollback. zoo.cfg tuning and security settings are carried. |
| كافكا | Log directories, including __consumer_offsets, stay in place. Offsets, topic overrides, ACLs, and the topic inventory are recorded for parity checks. ZooKeeper and KRaft modes are detected, and broker IDs and listeners are kept. |
| Solr | Cores stay in place and their replicas are activated under the new service. The Atlas index is never rebuilt before finalize. |
| Atlas | The graph tables stay in HBase. HBase snapshots are taken as a recovery seed and removed at finalize. |
| Oozie | The same database is re-pointed. The new share library is installed next to the existing one, which is kept until finalize, and customer jars in it are checked for a counterpart. |
| Hue | The database is re-pointed, and the Hue secret key is carried so that user sessions survive. |
| Ranger and Ranger KMS | The databases are re-pointed, and policies are exported before the cutover. Finalize refuses until every encryption zone key is proven decryptable on the target. |
Identities and Certificates
Kerberos
- Keytabs are copied, never regenerated. Phase 3 stages the keytabs from the Cloudera Manager process directories. The takeover places them under
/etc/security/keytabs/with the file names the stack expects, such asnn.service.keytabوspnego.service.keytab. Shared keytabs are split withktutil, which keeps the key material and the key version number byte for byte. Existing files are moved aside, never overwritten. - Key versions are recorded. Discovery records the principal, host, and key version number of every keytab, so that a change is detected.
- Ambari does not manage identities. Ambari Kerberos is enabled in a mode that asks the KDC or Active Directory for nothing:
kerberos-env:
kdc_type: none
manage_identities: "خطأ"
manage_auth_to_local: "خطأ"
install_packages: "خطأ"
create_ambari_principal: "خطأ"
krb5-conf:
manage_krb5_conf: "خطأ"
Principals, key versions, auth_to_local rules, and /etc/krb5.conf stay as they are. Only service identities are carried; the principals Cloudera Manager used for its own monitoring are left out. The HTTP authentication signer secret, which Cloudera Manager kept per role, is replaced by one cluster-wide /etc/security/http_secret.
TLS
- Certificates are reused. The estate's keystores and truststores, including those issued by Cloudera Manager Auto-TLS, are copied to
/etc/security/serverKeys/. Configuration values that pointed at the Cloudera Manager agent certificate directory or the{{CM_AUTO_TLS}}macro are re-pointed there. - Trust is checked. Whether each JDK in use trusts the CA of the estate is checked and reported.
- Renewal. Certificates are renewed later with
bifrost classic renew-certificatesfrom PEM, JKS, or PKCS12 material issued by the corporate PKI, or with the certificate management of Apache Ambari (Ilum build). See Day-2 operations. - Ranger plugins. The truststore, credential file, and audit keytab paths of every Ranger plugin are moved off the process directories.
Clients and Applications
| Item | After the migration |
|---|---|
| Hostnames and service ports | Unchanged. Every port a source role listened on is checked on the same host after the cutover. |
Principals and auth_to_local rules | Unchanged. |
HDFS nameservice, ZooKeeper quorum, fs.defaultFS | Taken from discovery, not from stack defaults. |
| Operating system users, groups, UIDs, GIDs | Unchanged. Ambari never creates or modifies them. |
| Gateway hosts | Cloudera Manager gateway roles become Ambari client components. bifrost classic refresh-clients rewrites /etc/<component>/conf from the desired configuration. |
| Scripts that call parcel paths | ال bin و lib entries of every active parcel are re-linked into the new installation, so /opt/cloudera/parcels/<product>/bin/<tool> keeps resolving. |
| Hive command line | ال خلية command uses Beeline, as on CDP. |
| Automation on the Cloudera Manager API | Re-pointed to the Ambari REST API, or served by the Cloudera Manager API compatibility layer of Apache Ambari (Ilum build). With ambari_cm_endpoint_takeover: true, Ambari also answers on the Cloudera Manager ports 7180 and 7183. |
Spark commands keep their meaning:
| Command | After the migration |
|---|---|
spark3-submit, spark3-shell, pyspark3 | Spark 3, where the source ran the Cloudera Spark 3 parcel |
شرارة تقديم, spark-shell, بايسبارك | Spark 2, where the source ran the Spark 2 included in CDP |
spark2-submit | Spark 2, as an additional name |
Common Pitfalls and How They Are Handled
- Cloudera paths in configuration. Refused before the cutover and checked again by
validate-post. Parcel paths used by scripts keep resolving through the re-linked client entries. - Classes missing from the target build.
bifrost classic scan-class-coveragelists every class referenced in configuration that the target jars do not contain, before the window. HBase coprocessor classes are included. The takeover stops on a missing class instead of starting a daemon that fails later. - Customer jars. Hive hooks, SerDes, and UDF jars are carried to a Bifrost-managed directory on
HIVE_AUX_JARS_PATH. Kafka plugins and the Oozielibextdirectory are carried the same way. - Queues edited in Queue Manager. When queue definitions live in the ZooKeeper mutation store, the live store is compared with the captured
capacity-scheduler.xml, and any difference stops the run before the services are stopped. Queues are then managed in the YARN Queue Manager view. - Cloudera lineage listeners. Navigator and other Cloudera lineage listeners are removed from the Spark configuration; the Atlas Spark hook is kept wherever Atlas runs.
- Process limits. Every daemon gets at least the
nprocوnofilelimits it had on Cloudera, measured from/proc/<pid>/limits, instead of the lower stack defaults. - Log locations. Log directories are carried. After the cutover, the open files of every daemon are checked to confirm that nothing writes into the old installation tree.
- Port collisions. Ambari serves on port 8085 by default, because port 8080 is used by the ZooKeeper admin server. Ports that differ between Cloudera and the stack defaults, such as the HBase Thrift port, are carried.
- Dependencies between services. Plan validation refuses a plan in which a migrated service still points at a service that is not part of the migration.
- Java. Components run on JDK 11 by default. The package swap stops if no JDK 11 is available to keep
/usr/bin/javaworking.
Replication
Replication Manager schedules move with the cluster:
- Capture. The replication peers, the schedules of every service, the snapshot policies, and the external accounts are captured at discovery and again in phase 3.
- Import. The takeover registers the peers and accounts in Apache Ambari (Ilum build) and creates the HDFS schedules (from a peer cluster, between clusters of the same Cloudera Manager, and to Amazon S3) and the Hive schedules (from a peer cluster). Every schedule is created paused. The schedules keep the Cloudera field names, so no schedule is rewritten by hand.
- Decisions. Where a schedule moved deleted files to the trash, the owner chooses per schedule whether files are deleted or kept.
- Activation. After validation,
bifrost classic replication-activate --executeruns a dry run of each schedule, resumes only the schedules whose dry run succeeded, and confirms that Ambari has scheduled the next run. Schedules that were paused in Cloudera Manager stay paused, and one-time schedules are not activated.
Peer credentials are kept in a credential store on the Ambari server; cloud credentials are kept in a credential store in HDFS.
Apache Ambari (Ilum Build)
Apache Ambari (Ilum build) extends upstream Apache Ambari with the operations that Cloudera Manager users rely on. It manages the services added by the Ilum Bigtop distribution, such as Impala, Atlas, Hue, Oozie, and Spark 2, as first-class services.
Replication Manager
- A Replication menu with replication schedules, snapshot policies, and replication peers.
- HDFS replication between clusters, including pulling from a peer managed by Ambari or by Cloudera Manager, with high availability, cross-realm Kerberos, and encryption zones.
- HDFS replication to cloud object storage: Amazon S3 and Azure Data Lake Storage.
- Hive replication of tables and data from a peer, with functions reloaded on every HiveServer2.
- Scheduling by cron expression or by interval, with pause and resume, run now, dry run, abort, and a run history with the copy counters of each run.
- Snapshot policies for HDFS and HBase, with daily, weekly, and monthly retention.
- Alerting on failed and aborted runs.
- Safe copies. The copy engine supports Cloudera-style path rebasing and closes upstream cases in which the delete option removes data unexpectedly.
Component Operations
| خدمة | Operations |
|---|---|
| HDFS | Roll edits, enter and leave safe mode, save namespace, rebalance, decommission, bootstrap the standby NameNode |
| غزل | Reconfigure the ResourceManager, refresh queues, format the state store, clean NodeManager recovery and local directories |
| HBase | Enable and disable the balancer, create the Ranger repository |
| خلية | Validate and upgrade the metastore schema |
| Any role | Thread dump, heap histogram, heap dump, and open files, from the host component menu |
Workloads are managed from the service pages: the YARN Applications tab lists and kills applications, and the Impala Queries tab lists and cancels queries. Diagnostic bundles can be collected for support.
Restarts and Availability
- Rolling restart with batches, health gates, and a failure budget. High-availability pairs are never restarted in the same batch, and the HDFS edit log is rolled after each JournalNode batch.
- Restart review that shows the reason for every pending restart, and refresh instead of restart where a configuration change can be applied live.
- Automatic actions that restart a component when an alert fires, with a rate limit.
- Role migration between hosts and a graceful stop of all services on a host.
- High availability for the NameNode, ResourceManager, Impala statestore and catalog, Hive Metastore, Oozie, Atlas, Hue, Livy, and Ranger UserSync and TagSync.
Monitoring and Logs
- بروميثيوس as the metrics source, built in or external, with
remote_writeto another store. - لوحات معلومات Grafana for the cluster and its services, and metrics tabs for Hive, ZooKeeper, Impala, Oozie, Ranger, Ranger KMS, and Spark. HDFS, YARN, and Kafka show the charts Cloudera Manager users know.
- Log search across the cluster with Loki.
- Alert definitions for TLS certificate expiry, replication failures, agent recovery, the Ranger audit spool, HBase regions in transition and compaction queues, and more, with heatmaps colored by alert thresholds.
- Alert targets for email, SNMP, and webhooks.
Security and Governance
- Certificate management: generate a CA, issue, rotate, and place certificates; certificates the cluster already serves are recorded and tracked; expiry raises an alert.
- Directory access: scheduled LDAP synchronization, group-to-role mapping, and custom roles.
- Governance page that shows whether Ranger enforcement, audit delivery, lineage, and tag synchronization work.
- Audit trail of management operations, with a dedicated permission to view it.
اتمته
- An OpenAPI specification with an API explorer, and a Python SDK.
- A batch API that applies several operations in one call.
- A Cloudera Manager API compatibility layer for reads, configuration changes, and start, stop, and restart commands, so that existing automation can be moved step by step.
الخطوات التالية
- Follow the procedure in Classic migration.
- Read how the takeover works in Management plane, and how identities and keys are carried in Security and encryption.
- Check components and platforms in the Support matrix.
- To discuss a migration, contact [البريد الإلكتروني محمي].