One Object Store, Two Processing Engines: Building a Local Spark, Iceberg, and Oracle Autonomous AI Database Lab

Data lake modernization is rarely as simple as copying files from one location to another. Before data can be migrated, teams need to discover it, understand its structure, profile its quality, transform it into useful formats, and validate that the target accurately represents the source.

Apache Spark is well suited to large-scale file processing, while Apache Iceberg adds table-level capabilities such as schema evolution, snapshots, and time travel. Oracle Autonomous AI Database provides a converged platform for relational, JSON, vector, spatial, graph, and analytical workloads.

The interesting question is not which platform should replace the other. It is how they can participate in the same modernization workflow without building unnecessary file copies or host-mounted filesystem bridges.

To explore that pattern, I integrated two of my container management scripts:

sparkIcebergManager.sh 1.2.0 manages a local Spark, Iceberg, MinIO, and Jupyter environment.
orclADBPodman.sh 1.4.0 manages an Oracle Autonomous AI Database Free container and its connection to the same MinIO object store.

The result is a local development environment in which Spark writes Iceberg and Parquet data to MinIO while Oracle accesses those same objects through DBMS_CLOUD over a secured S3-compatible endpoint.

This environment is intended for learning, development, and migration prototyping. It is not a local installation or emulator of the managed Databricks platform, and its development credentials should not be exposed to an untrusted network.

Why Build This Lab?

Many migration architectures introduce an intermediate filesystem, NFS mount, or set of manual copy steps between the data lake and the target database. Those workarounds make a demonstration possible, but they do not accurately represent an object-storage-centric data platform.

In this implementation, MinIO remains the authoritative data store:

                 Apache Spark + Iceberg
                  transform and manage
                           |
                           | S3 API
                           v
                  MinIO Object Storage
                     warehouse bucket
                           ^
                           | HTTPS + S3 API
                           |
             Oracle Autonomous AI Database
             query, load, validate, and export

Spark and Iceberg access MinIO directly on their private Compose network. Oracle joins a shared Podman network and reaches MinIO through an internal TLS proxy. No NFS layer or macOS host-directory workaround is required.

This supports several useful migration patterns:

Profile and transform lake data with Spark before loading it into Oracle
Query Parquet objects externally from Oracle before deciding whether to ingest them
Load curated files into native Oracle tables with DBMS_CLOUD.COPY_DATA
Export Oracle query results back to MinIO as Parquet for Spark processing
Compare source and target counts, aggregates, and representative records
Experiment with a hybrid design in which some data remains in the lake while curated workloads move into Oracle

What the Scripts Automate

The Spark manager starts the upstream Spark and Iceberg quickstart and adds the integration components required by Oracle:

Component	Purpose	Endpoint
Jupyter	PySpark and Scala notebooks	`http://localhost:8888`
Spark UI	Job and application monitoring	`http://localhost:8080`
Iceberg REST catalog	Iceberg catalog API	`http://localhost:8181`
MinIO API	Host access to S3-compatible storage	`http://localhost:9000`
MinIO Console	Object-store administration	`http://localhost:9001`
MinIO TLS proxy	Secured Oracle-container access	`https://warehouse.minio:443`

It also creates the shared oracle-datalake network, the persistent spark-iceberg-minio-data volume, the warehouse bucket, and a local certificate authority for the internal HTTPS endpoint.

The Oracle manager then:

Starts the myadb Autonomous AI Database Free container on oracle-datalake
Resolves the internal minio and warehouse.minio names
Copies the generated MinIO CA from the minio-s3-tls container
Updates the Oracle container operating-system trust store
Synchronizes the CA with the SSL wallet used by DBMS_CLOUD
Creates or refreshes the matt database user
Adds the network ACL for warehouse.minio
Creates the DATALAKE_S3_CRED credential
Calls DBMS_CLOUD.LIST_OBJECTS as the final database-level validation

Start the Spark and Iceberg Environment

Both environments must use the same container runtime. Because the Oracle script uses Podman, I explicitly select Podman when running the Spark manager.

First, validate the prerequisites and install the upstream quickstart:

			
chmod +x sparkIcebergManager.sh
CONTAINER_ENGINE=podman ./sparkIcebergManager.sh doctor
CONTAINER_ENGINE=podman ./sparkIcebergManager.sh install

Start the complete Spark, Iceberg, and MinIO stack:

CONTAINER_ENGINE=podman ./sparkIcebergManager.sh start

Verify the services and the Oracle-facing S3 endpoint:

			
CONTAINER_ENGINE=podman ./sparkIcebergManager.sh health
CONTAINER_ENGINE=podman ./sparkIcebergManager.sh tests3
CONTAINER_ENGINE=podman ./sparkIcebergManager.sh s3info

The tests3 command launches a temporary MinIO client on oracle-datalake and verifies HTTPS connectivity, S3 authentication, and access to the warehouse bucket.

Create an Iceberg Table with Spark

With the stack running, drop any previous copy of the Iceberg table:

			
podman exec spark-iceberg spark-sql -e "
CREATE NAMESPACE IF NOT EXISTS demo.blog;
DROP TABLE IF EXISTS demo.blog.dbms_cloud_events;
"

Then clear the previous sample object prefix and create a fresh table:

			
podman exec minio mc alias set local http://localhost:9000 admin password
podman exec minio mc rm --recursive --force 
    local/warehouse/blog/dbms_cloud_events
podman exec spark-iceberg spark-sql -e "
CREATE NAMESPACE IF NOT EXISTS demo.blog;
CREATE TABLE demo.blog.dbms_cloud_events (
  event_id BIGINT,
  source STRING,
  event_ts TIMESTAMP,
  amount DECIMAL(10,2)
) USING iceberg;
INSERT INTO demo.blog.dbms_cloud_events VALUES
  (1, 'orders',  TIMESTAMP '2026-08-21 15:45:00', 42.50),
  (2, 'returns', TIMESTAMP '2026-08-21 15:46:00',  9.99),
  (3, 'orders',  TIMESTAMP '2026-08-21 15:47:00', 78.25);
SELECT *
  FROM demo.blog.dbms_cloud_events
 ORDER BY event_id;
"

		

Run the following to add 10,000 random records:

			
podman exec spark-iceberg spark-sql -e "
INSERT INTO demo.blog.dbms_cloud_events
SELECT 
  id AS event_id,
  IF(id % 2 = 0, 'orders', 'returns') AS source,
  TIMESTAMP '2026-08-21 15:45:00' + CAST(id AS INTERVAL SECOND) AS event_ts,
  CAST(10.00 + (id % 100) * 0.50 AS DECIMAL(10,2)) AS amount
FROM (
  SELECT id 
  FROM (SELECT explode(sequence(1, 10000)) AS id)
);
SELECT COUNT(*) FROM demo.blog.dbms_cloud_events;
"

		

Spark writes the table’s Parquet data and Iceberg metadata into MinIO. The object layout resembles the following:

			
blog/dbms_cloud_events/data/00000-...parquet
blog/dbms_cloud_events/data/00001-...parquet
blog/dbms_cloud_events/metadata/00000-...metadata.json
blog/dbms_cloud_events/metadata/00001-...metadata.json
blog/dbms_cloud_events/metadata/...avro

		

The files can also be inspected through the MinIO Console:

CONTAINER_ENGINE=podman ./sparkIcebergManager.sh minio

The local development credentials are:

			
Access key: admin
Secret key: password
Bucket:     warehouse

Start Oracle Autonomous AI Database Free

In the Oracle script directory, start the database container:

			
chmod +x orclADBPodman.sh
./orclADBPodman.sh start

The start action creates or reuses oracle-datalake, attaches myadb, synchronizes the MinIO CA when the Spark stack is available, and tests container-level connectivity.

The database can take longer than the script’s initial countdown to finish opening the myatp service, especially on the first run when the ADB-Free image downloads and restores its seed PDB. The datalakeDB command waits for SQL connectivity before creating users or credentials.

You can validate DNS and HTTPS independently:

			
./orclADBPodman.sh datalakeInfo
./orclADBPodman.sh datalakeTest

These checks confirm that the Oracle container resolves minio and warehouse.minio, trusts the generated CA at the operating-system level, and can establish HTTPS connections to the proxy.

Configure and Validate `DBMS_CLOUD`

Complete the database-side integration with one command:

./orclADBPodman.sh datalakeDB

This is the decisive end-to-end test. The command creates the user, network ACL, credential, and wallet trust required by DBMS_CLOUD, and then executes the equivalent of:

			
SELECT object_name,
       bytes
  FROM DBMS_CLOUD.LIST_OBJECTS(
           'DATALAKE_S3_CRED',
           's3://warehouse.minio/'
       )
 FETCH FIRST 20 ROWS ONLY;

		

A successful result includes the Parquet files and Iceberg metadata written by Spark. At this point, the integration has proven both directions of responsibility:

Spark and Iceberg can create and manage objects in MinIO
Oracle can securely authenticate to and inspect the same object store through DBMS_CLOUD

Query the Spark-Written Parquet Files from Oracle

Object listing confirms connectivity, but the more useful test is querying the data. Connect as the matt user:

./orclADBPodman.sh sqluser

Create an Oracle external table over the Parquet files in the Iceberg table’s data directory. Drop the demo table first so the example can be rerun:

			
BEGIN
    EXECUTE IMMEDIATE 'DROP TABLE dbms_cloud_events_ext';
EXCEPTION
    WHEN OTHERS THEN
        IF SQLCODE != -942 THEN
            RAISE;
        END IF;
END;
/

		

			
DECLARE
    l_file_uri_list CLOB;
BEGIN
    SELECT LISTAGG('s3://warehouse.minio/' || object_name, ',')
             WITHIN GROUP (ORDER BY object_name)
      INTO l_file_uri_list
      FROM DBMS_CLOUD.LIST_OBJECTS(
               'DATALAKE_S3_CRED',
               's3://warehouse.minio/'
           )
     WHERE object_name LIKE 'blog/dbms_cloud_events/data/%.parquet';
    DBMS_OUTPUT.PUT_LINE(l_file_uri_list);
    DBMS_CLOUD.CREATE_EXTERNAL_TABLE(
        table_name      => 'DBMS_CLOUD_EVENTS_EXT',
        credential_name => 'DATALAKE_S3_CRED',
        file_uri_list   => l_file_uri_list,
        format          => JSON_OBJECT('type' VALUE 'parquet'),
        column_list     =>
            'EVENT_ID NUMBER,
             SOURCE   VARCHAR2(30),
             EVENT_TS TIMESTAMP,
             AMOUNT   NUMBER(10,2)'
    );
END;
/

		

Query the data directly from Oracle:

			
SELECT event_id,
       source,
       event_ts,
       amount
  FROM dbms_cloud_events_ext
 ORDER BY event_id;

		

This is useful for inspecting and validating Parquet data before loading it into the database. It is important, however, to understand what this example does: the external table reads the physical Parquet files under the Iceberg table’s data directory. It does not interpret Iceberg snapshots, delete files, or table history. Native Iceberg access should be used when Oracle must honor the Iceberg table’s complete metadata and transactional semantics.

Load the Curated Data into an Oracle Table

Once the external data has been validated, create a native target table:

			
BEGIN
    EXECUTE IMMEDIATE 'DROP TABLE dbms_cloud_events PURGE';
EXCEPTION
    WHEN OTHERS THEN
        IF SQLCODE != -942 THEN
            RAISE;
        END IF;
END;
/

		

			
CREATE TABLE dbms_cloud_events (
    event_id NUMBER,
    source   VARCHAR2(30),
    event_ts TIMESTAMP,
    amount   NUMBER(10,2)
);

		

For this small example, load it from the external table:

			
INSERT INTO dbms_cloud_events (
    event_id,
    source,
    event_ts,
    amount
)
SELECT event_id,
       source,
       event_ts,
       amount
  FROM dbms_cloud_events_ext;
COMMIT;

		

For a direct object-to-table load, use DBMS_CLOUD.COPY_DATA:

			
DECLARE
    l_file_uri_list CLOB;
BEGIN
    SELECT LISTAGG('s3://warehouse.minio/' || object_name, ',')
             WITHIN GROUP (ORDER BY object_name)
      INTO l_file_uri_list
      FROM DBMS_CLOUD.LIST_OBJECTS(
               'DATALAKE_S3_CRED',
               's3://warehouse.minio/'
           )
     WHERE object_name LIKE 'blog/dbms_cloud_events/data/%.parquet';
    DBMS_CLOUD.COPY_DATA(
        table_name      => 'DBMS_CLOUD_EVENTS',
        credential_name => 'DATALAKE_S3_CRED',
        file_uri_list   => l_file_uri_list,
        format          => JSON_OBJECT('type' VALUE 'parquet')
    );
END;
/

		

Use either the INSERT ... SELECT example or COPY_DATA for the same target data—not both—unless duplicate handling is part of the test.

Validate the Migration

A migration workflow should validate more than whether the load completed. Compare the external and native representations:

			
SELECT 'EXTERNAL' AS data_source,
       COUNT(*) AS row_count,
       SUM(amount) AS total_amount
  FROM dbms_cloud_events_ext
UNION ALL
SELECT 'ORACLE' AS data_source,
       COUNT(*) AS row_count,
       SUM(amount) AS total_amount
  FROM dbms_cloud_events;

		

For the sample data, both sides should return three rows and a total amount of 130.74.

A production migration would extend validation to include:

Counts by source partition or business date.
Minimum and maximum timestamps.
Null and distinct-value comparisons.
Business aggregates and control totals.
Checksums or hashes where data types permit stable comparison.
Representative record-level comparisons.
Logging through USER_LOAD_OPERATIONS for DBMS_CLOUD loads.

Export Oracle Data Back to MinIO

The integration is not limited to loading data into Oracle. DBMS_CLOUD.EXPORT_DATA can write Oracle query results back to the same MinIO bucket. In this ADB-Free container lab, CSV export works without additional database-side conversion helpers:

podman exec minio mc rm --recursive --force local/warehouse/oracle-export

			
BEGIN
    DBMS_CLOUD.EXPORT_DATA(
        credential_name => 'DATALAKE_S3_CRED',
        file_uri_list   =>
            's3://warehouse.minio/oracle-export/dbms_cloud_events_',
        format          => JSON_OBJECT('type' VALUE 'csv', 'header' VALUE true),
        query           => q'[
            SELECT event_id,
                   source,
                   event_ts,
                   amount
              FROM dbms_cloud_events
        ]'
    );
END;
/

		

Verify the exported object from the MinIO side:

			
podman exec minio sh -lc 
  "mc ls local/warehouse/oracle-export && 
   mc find local/warehouse/oracle-export --name '*.csv' --exec 'mc cat {}'"

This provides a useful pattern for publishing Oracle-curated reference data, generating downstream datasets, or reconciling database results with lake processing. Raw Spark reads from MinIO require Hadoop S3A support in the Spark image; the stock quickstart image used by this lab does not include org.apache.hadoop.fs.s3a.S3AFileSystem, so direct s3a:// CSV or Parquet reads are outside the base script flow.

The TLS Detail That Matters

One of the most important lessons from building this integration is that container-level TLS success does not automatically mean DBMS_CLOUD will trust the same endpoint.

After the generated CA was installed in the Oracle container operating-system trust store, curl could reach MinIO successfully. DBMS_CLOUD.LIST_OBJECTS, however, still required the CA in the database’s SSL wallet.

For this local ADB-Free lab, orclADBPodman.sh synchronizes the wallet at:

/u01/app/oracle/wallets/ssl_wallet

In its default local-demo configuration, the script backs up the original wallet before replacing it with an auto-login wallet containing the MinIO CA. This is intentionally a development convenience, not a general production wallet-management recommendation.

To manage the database wallet manually, bypass the automated replacement:

			
ORACLE_DATALAKE_DBMS_CLOUD_WALLET_MODE=skip 
    ./orclADBPodman.sh datalakeDB

If the existing wallet password is available, the script can add the CA without replacing the wallet by using ORACLE_SSL_WALLET_PASSWORD and a non-replacement wallet mode.

Restarting and Cleaning Up

The two stacks can be stopped independently:

			
./orclADBPodman.sh stop
CONTAINER_ENGINE=podman ./sparkIcebergManager.sh stop

Restart Spark first, followed by Oracle, and then repeat the database-level validation:

			
CONTAINER_ENGINE=podman ./sparkIcebergManager.sh start
./orclADBPodman.sh start
./orclADBPodman.sh datalakeDB

The Spark manager preserves the MinIO named volume and the shared external network when removing its Compose containers. If the Spark manager regenerates its TLS certificates, run datalakeDB again so the Oracle container and DBMS_CLOUD wallet trust the new CA.

From a Local Lab to a Migration Pattern

This implementation is intentionally small, but the pattern extends beyond a laptop.

Spark can remain the high-volume processing layer for discovering, profiling, and transforming data lake files. Iceberg can continue to manage datasets that benefit from an open table format. Oracle Autonomous AI Database can externally inspect the curated objects, load the data that belongs in the converged database, validate migration outcomes, and publish results back to object storage.

More importantly, the object store remains the stable integration boundary. That reduces host-specific workarounds and makes each stage independently testable:

Spark proves that it can create the expected objects.
The shared network and TLS proxy prove secure transport.
DBMS_CLOUD.LIST_OBJECTS proves database authentication and access.
The external table proves Oracle can interpret the data files.
The native load and reconciliation queries prove the migration result.

That is the foundation of a useful modernization workflow: process data with the engine best suited to the task, keep the integration boundary open and observable, and validate every transition before decommissioning the source.

One Object Store, Two Processing Engines: Building a Local Spark, Iceberg, and Oracle Autonomous AI Database Lab

Why Build This Lab?

What the Scripts Automate

Start the Spark and Iceberg Environment

Create an Iceberg Table with Spark

Start Oracle Autonomous AI Database Free

Configure and Validate `DBMS_CLOUD`

Query the Spark-Written Parquet Files from Oracle

Load the Curated Data into an Oracle Table

Validate the Migration

Export Oracle Data Back to MinIO

The TLS Detail That Matters

Restarting and Cleaning Up

From a Local Lab to a Migration Pattern

Resources

Like this:

Related

Leave a ReplyCancel Reply

Why Build This Lab?

What the Scripts Automate

Start the Spark and Iceberg Environment

Create an Iceberg Table with Spark

Start Oracle Autonomous AI Database Free

Configure and Validate DBMS_CLOUD

Query the Spark-Written Parquet Files from Oracle

Load the Curated Data into an Oracle Table

Validate the Migration

Export Oracle Data Back to MinIO

The TLS Detail That Matters

Restarting and Cleaning Up

From a Local Lab to a Migration Pattern

Resources

Share this:

Like this:

Related

Leave a ReplyCancel Reply

Discover more from OraMatt: YABAOracle

Configure and Validate `DBMS_CLOUD`