Sunday, November 16, 2014

Finding table with Max SPU disk

Often time, when you issue a nz_storage_stats command or from nzadmin GUI under NPS, you noticed the following info

 Minimun | Average | Maximum
---------+---------+---------
 42.591  | 43.241  | 49.801


You can get the output above from the query below:

        SELECT
                TO_CHAR ( min_disk_usage , 'FM999,999,999,990.000' ),
                TO_CHAR ( avg_disk_usage , 'FM999,999,999,990.000' ),
                TO_CHAR ( max_disk_usage , 'FM999,999,999,990.000' )
        FROM
        (
       SELECT
               100 * MIN ( sds_used::numeric(38,10) / (_v_spudevicestate.sds_size::bigint * (1024^2))  ) AS min_disk_usage,
               100 * AVG ( sds_used::numeric(38,10) / (_v_spudevicestate.sds_size::bigint * (1024^2))  ) AS avg_disk_usage,
               100 * MAX ( sds_used::numeric(38,10) / (_v_spudevicestate.sds_size::bigint * (1024^2))  ) AS max_disk_usage
       FROM    _v_dslice
               left outer join _v_spudevicestate on
               ( _v_dslice.ds_prihwid  = _v_spudevicestate.sds_hwid AND
                 _v_dslice.ds_priptid  = _v_spudevicestate.sds_device )
        )
        AS the_subselect;

When, you have this info, how do you find out which table having this Max SPU disk usage.  Below you can determine the table name for the Max data slice.

with
GetMaxDSlice (DS_ID, DS_PCT, ROW_NUM) as
(select ds.ds_id as DS_ID,
(((sp.EXTENTS_USED * 3)/(sp.EXTENT_COUNT * 3)::float)*100)::numeric(10,2) as DS_PCT,
row_number () over (order by (((sp.EXTENTS_USED * 3)/(sp.EXTENT_COUNT * 3)::float)*100)::numeric(10,2) desc ) as Row_Num
FROM _v_dslice ds  LEFT OUTER JOIN _v_spudevicemap prm on (ds.ds_prihwid = prm.sdm_spuhwid AND ds.ds_priptid = prm.sdm_spudevice AND prm.sdm_role = 0)  LEFT OUTER JOIN _v_spudevicemap mir on (ds.ds_prihwid = mir.sdm_spuhwid AND ds.ds_priptid = mir.sdm_spudevice AND mir.sdm_role = 1)  LEFT OUTER JOIN _v_hwcomp hwp on (prm.sdm_sdhwid = hwp.hw_hwid)  LEFT OUTER JOIN _v_hwcomp hwm on (mir.sdm_sdhwid = hwm.hw_hwid)  LEFT OUTER JOIN _vt_disk_partition sp on (ds.ds_prihwid = sp.HWID AND ds.ds_priptid = sp.DEVICE AND sp.PART_NAME='PART_A')
),
GetTableId (TBL_ID, DS_ID, Row_Num) as
(select 
 tblid as TBL_ID,
 dsid as DS_ID,
 row_number () over (order by used_bytes desc) as row_num
 from _vt_spu_table_info
),
GetTableName (Table_Name, Obj_Id) as
(select 
 objname as Table_Name, objid as Obj_Id
 from _v_obj_relation_xdb
)
select GetTableName.Table_Name AS TableNameWithMaxUsageInDS,
  GetMaxDSlice.ds_id AS DSWithMaxUsagePCT,
  ds_pct AS PCTMaxForDS
from GetMaxDSlice, GetTableId, GetTableName
where GetMaxDSlice.Row_Num=GetTableId.Row_Num
and GetTableId.TBL_ID=GetTableName.obj_id
and GetMaxDSlice.Row_Num=1;


     TABLENAMEWITHMAXUSAGEINDS      | DSWITHMAXUSAGEPCT | PCTMAXFORDS
------------------------------------+-------------------+-------------
 PREPAID_MSG_SERVICE_USAGE_BKP_0730 |                48 |       49.80

Tuesday, July 8, 2014

Check Netezza Active Sessions/Queries

There are different ways to find active sessions in Netezza, portal or nzsession command line.  But if you like SQL script style command, below is the script:


select  q.qs_sessionid,
        q.qs_planid,
        q.qs_clientid,
        s.dbname,
        s.username,
        s.TYPE SESSION_TYPE,
        s.STATUS SESSION_STATUS,
        s.PRIORITY SESSION_PRIORITY,
        q.qs_cliipaddr,
        q.QS_STATE,
        substr(q.qs_sql,1,200),
        case
            when q.qs_state = 1 then 'pending'
            when q.qs_state = 2 then 'queued'
            when q.qs_state = 3 then 'running'
            when q.qs_state = 4 then 'aborted'
            when q.qs_state = 5 then 'done'
            else 'unknown'
        end QS_STATE_DESCRIPTION,
        q.qs_tsubmit+t.tzoffset qs_tsubmit,
        q.qs_tstart+t.tzoffset tzoffset,
        case when q.qs_tstart = t.EPOCH then 0 else abstime 'now' - q.qs_tstart end running_duration,
        q.qs_pritxt,
        q.qs_estcost,
        q.qs_estdisk,
        q.qs_estmem,
        q.qs_snippets,
        q.qs_cursnipt,
        q.qs_resrows,
        q.qs_resbytes,
        to_timestamp('08JUL14224216','DDMONYYHH24MISS')
from    _v_qrystat q,
        _v_session s,
                _vt_pg_time_offset t
where   q.qs_sessionid = s.id

Sample Output:

 QS_SESSIONID | QS_PLANID | QS_CLIENTID |       DBNAME       |     USERNAME     | SESSION_TYPE | SESSION_STATUS | SESSION_PRIORITY | QS_CLIIPADDR | QS_STATE |                                                                                                  SUBSTR                                                                                                  | QS_STATE_DESCRIPTION |     QS_TSUBMIT      |      TZOFFSET       | RUNNING_DURATION | QS_PRITXT | QS_ESTCOST | QS_ESTDISK | QS_ESTMEM | QS_SNIPPETS | QS_CURSNIPT | QS_RESROWS | QS_RESBYTES
--------------+-----------+-------------+--------------------+------------------+--------------+----------------+------------------+--------------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------+---------------------+------------------+-----------+------------+------------+-----------+-------------+-------------+------------+-------------
        59376 |     17368 |         112 | CELENR_HLD_PROD_DB | CELEBI_USR_PETL  | sql          | active         |                3 | 10.10.62.15  |        3 | SELECT COUNT(*) FROM ( SELECT CBP_NODE,(((cast(END_SEQ_NUMBER as bigint)-cast(start_seq_number as bigint))+1)-COUNT(*)) AS DIFF FROM HLD_NGIN_ADT_RPT_REC WHERE SEQ_CHECK IS NULL GROUP BY 1,END_SEQ_NUM | running              | 2014-07-09 11:53:37 | 2014-07-09 11:53:37 |                0 | normal    |         26 |          0 |         0 |           2 |           1 |          0 |           0
        56627 |     17288 |         211 | CELENR_HLD_PROD_DB | CELEBI_TEMP_ETL  | sql          | active         |                3 | 10.10.62.15  |        3 | insert into CELENR_HLD_PROD_DB..temp0_final_xfn_MSC select tmp.row_id, ph_calling.electronic_device_type_id calling_electronic_device_type_id, ph_called.electronic_device_type_id called_electronic_dev | running              | 2014-07-09 11:48:20 | 2014-07-09 11:48:21 |              316 | normal    |     150160 |      57811 |     91377 |          19 |          12 |          0 |           0
        58355 |     16828 |         254 | CELEBI_ISD_DB      | CELEBI_USR_08732 | sql-odbc     | active         |                3 | 10.10.47.229 |        3 | SELECT * FROM CELEBI_ZOOM_DB.EBINZADMIN.ZA_POSTPAID_VOICE_2G_3G A LEFT JOIN (SELECT ACCOUNT_ARR_ID,MDN, BILLED_TOTAL_INVOICED_AMOUNT from CELEBI_SOR_PROD_DB..INVOICE X inner JOIN (SELECT SUBSCRIBER_AR | running              | 2014-07-09 11:33:16 | 2014-07-09 11:33:16 |             1221 | normal    |      10621 |      31101 |     25983 |           6 |           6 |   49484266 | 10583460440
        59070 |     17361 |         374 | CELENR_STG_PROD_DB | CELEBI_USR_PETL  | sql          | active         |                3 | 10.10.62.15  |        3 | UPDATE STG_NGIN_CDR_SMS stg SET DESTINATION_SERVICE_PROVIDER_ID=USG.SERVICE_PROVIDER_ID FROM tmp_USAGE_POINT_SP_sms usg WHERE USG.POINT=SUBSTRING(stg.calledpartynumber, 1, 7) AND STG.Serviceflow =1 AN | running              | 2014-07-09 11:53:29 | 2014-07-09 11:53:29 |                8 | normal    |         63 |        202 |         5 |      

Friday, June 27, 2014

nzbackup vs nz_backup

There are 2 commands to perform backup of Netezza.  One is nzbackup.  The other one is nz_backup.  For nzbackup, not much you can play with it except you need to specify the database to backup.  However, nz_backup provides more command options eg, specific table to backup, include where clause etc.  Here's the command options for nz_backup.


Usage:    nz_backup   -dir <dirname>  -format <ascii|binary|gzip>  [ optional args ]
               -or-
          nz_restore  -dir <dirname>  -format <ascii|binary|gzip>  [ optional args ]

Purpose:  To backup (or restore) one or more tables.

          An nz_backup must be run locally (on the NPS host being backed up).

          An nz_restore can be used to restore data into a remote NPS host.  Just
          include the "-host" switch, or set the NZ_HOST environment variable.

          Note:  When doing an "nz_restore -format binary -host <xxx>", the script
                 issues an "nzload -compress true".  This nzload feature only exists
                 in nzload as of NPS 4.6.  If you want to do this on an older version
                 of NPS (4.0 or 4.5) then:
                 o   Install a copy of the 4.6 client toolkit somewhere on your box
                     (it can be used against the older server releases)
                 o   Add it's bin directory to the start of your search PATH
                 o   Then invoke nz_restore

          These scripts can process a single table, multiple tables, or an entire database.

          The data format that is used can be either
               ascii  -- which is very portable.
               binary -- which is the database's compressed external table format.  This
                         is much faster, and results in significantly smaller backup sets.
               gzip   -- ascii, which is gzip'ed on the NPS host.

          The data can be written to (or read from) disk files or named pipes.
               If you use named pipes, another application is used to consume
               (or produce) the data.

          These scripts just concern themselves with the DATA itself.  When backing up
          a table, the DDL is not included.  When restoring a table, the script expects
          the table to already exist.  It will not create it.  It will not truncate it
          (so if the table currently has any data in it, that data will be left untouched
          by this script).

          To backup tables requires the following permissions:

               GRANT SELECT       ON  _VT_HOSTTXMGR         TO <user/group>;
               GRANT SELECT       ON  _VT_HOSTTX_INVISIBLE  TO <user/group>;
               GRANT SELECT       ON  _VT_DBOS_CONNECTION   TO <user/group>;
               --To obtain information about transactions

               GRANT LIST         ON  <DATABASE|dbname>     TO <user/group>;
               --The user must have access to the database that contains the tables

               GRANT SELECT       ON  <TABLE|tablename>     TO <user/group>;
               --The user must have access to the tables themselves, and their data

               GRANT CREATE EXTERNAL TABLE                  TO <user/group>;
               --The user must be able to create external tables, into which the
               --data will be unloaded

          To restore/reload a table requires an additional permission:

               GRANT INSERT       ON  <TABLE|tablename>     TO <user/group>;
               --The user must be able to insert (i.e., reload) data back into the tables

Options:  REQUIRED ARGUMENTS
          ==================

          -dir <dirname> [...]

               The full path to the directory in which the data files will be written
               to (or read from).  This directory must already exist and permit write
               access to it.  The directory name can be as meaningful as you wish to
               make it.

               If you are running this script as some linux user other than 'nz', please
               note that it will actually be one of the 'nz' processes that writes the
               data into this directory.  So linux user 'nz' must also have write access
               to it.  If you are using named pipes (rather than files) then this is the
               directory where the named pipes will be created.

               Examples:
                    -dir /backups/backup_set_17
                    -dir /snap_storage/bu/customer_db/2006_11_18
                    -dir /tmp

               If desired, you may split the backup files up across multiple directories/
               file systems.  Each thread can be associated with a separate "-dir <dirname>"
               by specifying them on the command line.  If you use this feature, then the
               number of directories specified must match the number of threads.

          -format <ascii|binary|gzip>

               Identifies the format to be used for the output files.

               ascii    Universal in nature, but typically results in larger files and
                        slower performance.

               binary   The database's compressed external table format.

               gzip     ascii, which is then compressed (using gzip).  By definition,
                        compressing and decompressing data uses up a lot of CPU cycles (i.e.,
                        it takes a long time).  When using the binary format (compressed/
                        external), the work is done in parallel across all of ths SPUs ... so
                        it is very quick.  But this option uses the NPS host to gzip/gunzip the
                        data.  You will (almost always) want to use multiple threads in order
                        to get more of the host's SMP processors involved in order to speed
                        things along.  The sweet spot seems to be about 8 threads, though you
                        can certainly use a larger/smaller number if you want to break the
                        backup files up into more/fewer pieces.

                                        best      --to--     least
                                       ======     ======     ======
                               Speed:  binary     ascii      gzip
                                Size:  gzip       binary     ascii
                        Universality:  ascii      gzip       binary


          OPTIONAL ARGUMENTS
          ==================

          -h or -? or --help       display this help

          -[rR]ev or -V            print the software revision of this program

          -host <name/IP>          host name or IP address  [NZ_HOST]
          -u    <user>             user name                [NZ_USER]
          -pw   <string>           password                 [NZ_PASSWORD]
          -db   <database>         database name            [NZ_DATABASE]

          -t <tablename> [...]

               # Table(s) within the database to be processed.  If none are specified,
               # then all tables in the database will be processed.
               #
               # If you have a file that contains a list of tablenames to be backed up,
               # (separated by spaces and/or newlines) it can be used via the following
               # syntax:
               #
               #      -t `cat /tmp/the_list_of_tables`

          -threads <nnn>

               # Breaking the backup into multiple threads (per table) can increase the
               # overall thruput, especially for large tables.  This will also have the
               # effect of creating smaller backup files, since each one will now be
               # broken up into multiple pieces.
               #
               # By default, only a single thread will be used.  You can specify a number
               # from 1..31.  Whatever value you specify for the backup must also be used
               # for the restore.  In general, the sweet spot seems to be about 6 threads.

          -script <scriptname>

               # Instead of backing up (or restoring) the data from disk files, you can use
               # named pipes -- allowing another application to consume (or produce) the data
               # on the fly.  To use named pipes, specify that application/script here.  The
               # script will be automatically invoked and passed (as arg 1) the fully rooted
               # pathname of the named pipe that is it supposed to use.
               #
               # For example scripts, see the file(s)
               #      nz_backup.script_example
               #      nz_restore.script_example

          -whereclause <clause>

               # Normally, nz_backup is used to backup the entire contents of a table --
               # all visible rows.  This option allows "you" to tack on a WHERE clause
               # to the data that gets selected ... allowing "you" to backup a subset
               # of the table.  All the power (and responsibility) is put into your
               # hands.  Do wrap the clause in double quotes so it will be passed into the
               # script correctly.  Examples:
               #
               #      -whereclause  "customer_key = 2"
               #      -whereclause  "customer_key in (1,3,5)"
               #      -whereclause  "region_name = 'AMERICA' or region_key = 0"
               #      -whereClause  "order_date between '1998-01-01' and '1998-12-31'"
               #
               # Because this clause gets applied to all tables being backed up, you would
               # probably only want to backup a single table at a time (when using this
               # clause) ... since the clause will typically contain column names that are
               # specific to that table.
               #
               # This clause only applies to backups (not restores).  Its use will be logged
               # in the output of this script (as well as in the pg.log file).

          -ignoreTxID

               # Tables are individually backed up, one at a time.  Since a backup may span
               # many hours, this script insures that the backup represents a consistent
               # point-in-time by using the transaction IDs attached to each row.
               #
               # This switch will override that feature ... and backs up each table with
               # whatever data it contains when the backup (of that particular table) is
               # kicked off.  This insures that your backup will include ALL of the data
               # in a table that has been committed (so you're not susceptible to long
               # running or stale transactions).
               #
               # This switch is primarily of use with NPS 4.0.  In later releases, this
               # script is able to do things differently.

          -dropdata

               # This is for testing purposes only.  As the name implies, the backup will
               # be written to /dev/null, resulting in no backup at all.  This is useful
               # for testing the performance of the NPS components that are involved
               # (SPUs/S-Blades ==> Host), while excluding the speed/overhead of your host
               # storage.

          -sizedata

               # This is for testing purposes only.  Like "-dropdata", but rather than
               # sending the backup data directly to /dev/null it will first be piped
               # thru "wc -c" in order to count the number of bytes in the backup
               # stream (e.g., to provide you with actual sizing information).  So it
               # has the performance characteristics of "-dropdata" ... but provides
               # you additional information.
               #
               # You can use "-format <ascii|binary>" and 1 or multiple "-threads <nn>"
               # when using this switch.
               #
               # Each table will include the following pieces of information
               #
               #      Info:  source table size       80,740,352
               #      Info:  backup file size        76,574,691
               #
               # And summary lines for the entire backup set will display the
               #
               #      TOTAL source table size :  48,267,526,144
               #      TOTAL backup file size  :  46,088,648,210
               #
               # The table size is whatever the table size is -- the amount of storage
               # space it using on disk (as reported by nz_db_size or nz_tables).  This
               # script doesn't know/care if the data is compressed on disk (e.g, CTA0/1/2).
               # Nor does the script know if there are any logically deleted rows in the
               # table (taking up space in the table, but which would not be part of a
               # backup data set).
               #
               # The backup size is the amount of storage that would be required if the
               # backup data set was actually written to disk.  This would represent either
               # the ascii or the binary (compressed external table format) version of the
               # data ... whatever you chose.
               #
               # To get sizing information for a full nzbackup, e.g.
               #      nzbackup   -db DBNAME  -dir /tmp
               # you would use a command line such as this
               #      nz_backup  -db DBNAME  -dir /tmp     -format binary  -sizedata

Outputs:  Status/log/timing information will be sent to standard out ... and will
          include information about any ERROR's that might be encountered.

          Exit status:  0 = success, non-0 = ERROR's were encountered

Examples: $ nz_backup  -format binary -dir /backupdir

          $ nz_restore -format ascii -dir /tmp -db my_db -t table1 -script /tmp/my_script

Comparison:               nzbackup/nzrestore     nz_backup/nz_restore
                          ==================     ====================
                           NPS CLI Utility          Add-on Script
Backup Type
  Full                            X                       X
  -differential                   X
  -cumulative                     X

Granularity
  Entire Database                 X                       X
  Individual Table(s)         nzrestore                   X

Formats Supported
  Compressed Binary               X                       X
  Ascii                                                   X
  Ascii gzip'ed                                           X

Output Destination
  Veritas (NPS 4.0)               X
  Tivoli  (NPS 4.6)               X
  Disk                            X                       X
  Named Pipes                                             X

Multi-Stream Support         nzbackup (6.0)               X

DDL included as part              X              Use the nz_ddl* scripts
  of the backup set

Friday, June 13, 2014

Netezza Versioned Tables

Versioned tables come about as a result of doing an
               ALTER TABLE <tablename> [ADD|DROP] COLUMN ...

          This results in multiple data stores for the table.  When you go to query
          the table, NPS must recombine the separate data stores back into a single
          entity.  This action will be performed automatically and on-the-fly.  But
          it does result in additional query overhead.  Therefore, it is a best
          practice to reconstitute the table as soon as practical by doing a

               GROOM TABLE <tablename> VERSIONS;

          Notes:  The maximum number of table versions allowed is 4 (which means you
                  can perform at most three ALTER TABLE commands before doing a GROOM,
                  since this number includes the original table version itself)


SELECT  the_table.database as "Database",
        MAX
        (   CASE WHEN the_table.objclass = 4910                THEN   SUBSTR(the_table.objname, 3)::nvarchar(128)
                 WHEN the_table.objclass in (4951,4959,4963)   THEN NULL
                 ELSE   the_table.objname::nvarchar(128)
            END
        ) AS "Table Name",
        TO_CHAR ( nvl(SUM ( used_bytes ),0), '999,999,999,999,999,999'  ) AS "Size (Bytes)",
count(distinct(case when objclass in (4959,4963) then objid else null end)) as "# Of Versions"
FROM    _V_OBJ_RELATION_XDB the_table
        left outer join _V_SYS_OBJECT_DSLICE_INFO on
        (
               the_table.objid = _V_SYS_OBJECT_DSLICE_INFO.tblid   and
               _V_SYS_OBJECT_DSLICE_INFO.tblid > 200000
        )
WHERE
            the_table.objclass in (4905,4910,4940,4959,4951,4953,4961,4963)
        and the_table.objid > 200000
GROUP BY
        "Database",
        the_table.visibleid
HAVING  "# Of Versions" >= 2
ORDER BY 1, 2;


Below is sample output:

# Of Versioned Tables         13
     Total # Of Versions      26

       Database        |                       Table Name                       |       Size (Bytes)       | # Of Versions
-----------------------+--------------------------------------------------------+--------------------------+---------------
 CELEBI_BSS_SIT_DDS_DB | SUBSCRIPTION_ACQUISITION_REVENUE_SUMMARY_FULL_11062014 |              566,755,328 |             2
 CELEBI_PREPROD_DDS_DB | COGNOS_CUBE_CONTROL                                    |                2,883,584 |             2
 CELEBI_PREPROD_DDS_DB | COGNOS_CUBE_CONTROL_HIST                               |                  262,144 |             2
 CELEBI_SOR_DB         | REV_GROUP_MASTER                                       |               24,117,248 |             2
 CELEBI_STG_DB         | STG_SAP_CSKU                                           |                  262,144 |             2

Wednesday, June 11, 2014

Check if statistics out of date

You can use query below to check if statistics are outdated.  From nzsql, connect to a database in particular and run the query below:

nzsql -d database <<eof

SELECT
        "Table Name",
        the_label as "Statistics State",
        count(*)  as "# Of Columns",
  TO_CHAR ( _v_table.reltuples, '999,999,999,999,999') AS "Table Rowcount"
FROM
(
SELECT
                _v_relation_column.name as "Table Name",
                CASE WHEN       _v_statistic.recent IS NULL THEN 'not maintained    '
                ELSE
                CASE WHEN       _v_statistic.recent =    85 THEN 'Full              '
                     WHEN       _v_statistic.recent =   341 THEN 'Express           '
                     WHEN       _v_statistic.recent =   149 THEN 'Basic             '
                     WHEN       _v_statistic.recent =     1 THEN 'Full Min/Max OK   '
                     WHEN       _v_statistic.recent =   257 THEN 'Express Min/Max OK'
                     WHEN       _v_statistic.recent =   129 THEN 'Basic Min/Max OK  '
                     WHEN       _v_statistic.recent =     0 THEN 'Full Outdated     '
                     WHEN       _v_statistic.recent =   256 THEN 'Express Outdated  '
                     WHEN       _v_statistic.recent =   128 THEN 'Basic Outdated    '
                     WHEN       _v_statistic.recent =   169 THEN 'Min/Max Only      '
                     WHEN       _v_statistic.recent =   170 THEN 'Unavailable       '
                     WHEN       _v_statistic.recent = 16554 THEN 'Unavailable       '
                     ELSE       _v_statistic.recent || ''
                     END
                END as the_label,
                _v_relation_column.objid
        FROM
                _v_relation_column
                left outer join _v_statistic on
                (   _v_relation_column.name   = _v_statistic.tablename   AND
                    _v_relation_column.attnum = _v_statistic.attnum
                )
        WHERE
                    _v_relation_column.objid > 200000
                AND upper(_v_relation_column.type) in ('TABLE','SECURE TABLE')
) the_subquery
 LEFT OUTER JOIN _v_table ON the_subquery."Table Name"  =  _v_table.tablename
WHERE trim(the_label) NOT IN ('not maintained','Full','Express','Basic','Unavailable')

GROUP BY "Table Name", "Statistics State", "Table Rowcount"
  ORDER BY "Table Name", "Statistics State";

eof

Tuesday, June 10, 2014

unload and load data using External Table

There a few ways you can transfer Netezza data across another Netezza database.  One way is to use nz_migrate.  Another way through Aginity Tools -> Import and the third way is through External Table in Aginity.  Note:  You can only use this method from Aginity

Below example, dumps the data from the query to your local PC temp directory while connecting to the source database

UNLOAD DATA

CREATE EXTERNAL TABLE ext_db 'c:\temp\ext_db.dat'
USING
(
remotesource 'odbc'
delimiter '|'
ignorezero false
ctrlchars true
escapechar '\'
logDir 'c:\temp'
boolStyle 'T_F'
encoding 'internal'
nullValue 'N'
maxErrors 1
)
AS
SELECT database, owner, cast(createdate as date) as ts  from _v_database;


While connecting your Aginity to your target NPS host database, this method can be used to load the data
to your say Development environment.

LOAD DATA (in different NPS host database)

create external table abc
(
       DATABASE NATIONAL CHARACTER VARYING(255),
       OWNER NATIONAL CHARACTER VARYING(255),
       TS DATE
)
using
(
dataobject ('c:\temp\ext_db.dat')
Remotesource 'odbc'
Encoding 'INTERNAL'
EscapeChar '\'
CrInString true
CtrlChars true
Delimiter '|'
NullValue 'NULL'
)

;

Thursday, June 5, 2014

Netezza Groups

There two types of groups in Netezza.  One type is the typical permission, privileges on specific objects in the NPS.  Another group is called Resource Sharing Group (RSG) for workload management.  You can use this group to define resource utilisation percentages from total NPS resources.

Object Permission Group

CREATE GROUP CELEBI_USR_PETL_ST_GRP;

Or, to create group and add user(s)

CREATE GROUP FIRST_GROUP WITH USER1,USER2;

or add user in group using alter

ALTER GROUP FIRST_GROUP ADD USER USER1;

or drop user from group

ALTER GROUP FIRST_GROUP DROP USER USER1;


RSG Group

CREATE GROUP EBI_ENR_USER WITH
SESSIONTIMEOUT 30
QUERYTIMEOUT 300
DEFPRIORITY NORMAL
MAXPRIORITY NORMAL
RESOURCE MINIMUM 10
RESOURCE MAXIMUM 70
MAXIMUM 30 ;

How to lock Netezza userid

How do you lock Netezza userid?  You actually cannot find any way from NZ Admin tool or Aginity or even from the portal.  Then, how do you lock a Netezza userid?  Below command will show you:


SYSTEM(ADMIN)=> update _t_user set uselocked=true where usename='TEST1';  
UPDATE 1  


If you want to unlock, you can use NZ Admin tool.

Wednesday, June 4, 2014

find who owns what objects

How to you find out who owns what in the Netezza databases.  You need to know this because sometimes you want to drop a particular object, NPS returns error like below:

DROP USER: user "BIADM" owns objects, cannot be removed

So how do you find out which objects belongs to BIADM?  

Here's how:

SELECT OWNER, THE_OBJECT_TYPE, OBJNAME
FROM (
SELECT
                usename AS OWNER,
                (CASE   WHEN objclass = 4902 then 'database'
                        WHEN objclass = 4903 then 'group'
                        WHEN objclass = 4904 then 'user'       END ) AS the_object_type,
                objname
        FROM
                _t_object,
                _t_user
        WHERE   objowner = usesysid
         AND    objclass IN (4902, 4903, 4904)
         AND    objdb = 0
         AND    objowner <> (SELECT usesysid FROM _t_user WHERE UPPER(usename) = 'ADMIN')
UNION ALL
        SELECT
                _t_user.usename AS OWNER,
                lower(the_class.objname) AS the_object_type,
                 the_object.objname || ' in database ' ||  the_database.objname AS objname
        FROM
                _t_object as the_object
                        LEFT OUTER JOIN _t_object the_class ON
                                (the_object.objclass = the_class.objid),
                _t_object the_database,
                _t_user
        WHERE   the_object.objowner = _t_user.usesysid
         AND    the_object.objclass NOT IN (4902, 4903, 4904)
         AND    the_object.objdb = the_database.objid
         AND    the_object.objowner <> (SELECT usesysid FROM _t_user WHERE UPPER(usename) = 'ADMIN')
) tab
WHERE OWNER='BIADM'
ORDER BY 1,2,3;

Tuesday, June 3, 2014

Netezza Admin / DBA role privileges

In Netezza, there is a default admin user which you can use.  But how about creating a second admin user?  You can assigned the permissions below.

create group DBA_GROUP;

-- Object permission
grant all on USER to DBA_GROUP;
grant all on MANAGEMENT VIEW to DBA_GROUP;
grant all on MANAGEMENT table to DBA_GROUP;
grant all on SYSTEM VIEW to DBA_GROUP;
grant all on SYSTEM TABLE to DBA_GROUP;
grant all on PROCEDURE to DBA_GROUP;
grant all on LIBRARY to DBA_GROUP;
grant all on AGGREGATE to DBA_GROUP;
grant all on FUNCTION to DBA_GROUP;
grant all on SYNONYM to DBA_GROUP;
grant all on SEQUENCE to DBA_GROUP;
grant all on EXTERNAL TABLE to DBA_GROUP;
grant all on DATABASE to DBA_GROUP;
grant all on GROUP to DBA_GROUP;
grant all on TABLE,VIEW to DBA_GROUP;

-- Admin permission
grant ALL ADMIN to DBA_GROUP;

create user NZADMIN with password 'abc123';

alter group DBA_GROUP add user NZADMIN;

create group DBA_GRA_GROUP with resource minimum 50 resource maximum 100;

alter user NZADMIN in resourcegroup DBA_GRA_GROUP;

Saturday, May 31, 2014

find user last login

Often times IT security would want to find out which DB users have been inactive for the last 6 months.  Below query will find out.

Note: You need Query History DB enabled to do this.

select username from _v_user
minus
select distinct sessionusername
from (
select sessionusername, to_char(connecttime,'MMYYYY') as connectime, count(*)
from "$hist_session_prolog_1"
group by sessionusername, to_char(connecttime,'MMYYYY')
having to_char(connecttime,'MMYYYY') in (122013,012014,022014,032014,042014,05014)) a

Stored Procedures vs UDX


Calling a stored procedure


The standard way to call a stored procedure is by using the CALL statement.
call myprocedure(arguments);

You can also invoke a stored procedure using a SELECT statement, but you can't include a FROM clause. So this works:
select myprocedure('foo', 'bar', 123);

This will throw an error:
select myprocedure(x,y,z) from foo;

It is also important to note that a stored procedure cannot be called as part of an expression, even from within NZPLSQL code. So the code below is also illegal and will result in an error.
select myprocedure(arguments) + 100;
-- or in NZPLSQL code
result := myprocedure(arguments) + 100;

Stored Procedure Performance

It is important to understand that a stored procedure runs as a single process on the NPS host, it does not run in parallel.  Any SQL statement issued by a stored procedure behaves just as SQL does when issued by any other source (the SQL runs in parallel on the NPS SPUs).  The primary use case for stored procedures is to control the execution of a multiple step process where all of the work done by each step is performed by a SQL statement.  A stored procedure can be used to dynamically generate a SQL statement based on the inputs to the procedure or the values in a table, but it should never be used to actually process large amounts of data returned from a SQL statement.  Any logic which needs to be applied to a large number of individual rows should be done in a SQL statement or a UDX, never by using IF statements in the procedure itself.  

Procedures vs. Functions

Procedures in Netezza are different than functions. A procedure can only be used as shown above (not with a FROM clause), but a procedure can run SQL. A function can be used with a FROM clause and can process many rows of data instead of only one, but a function cannot issue SQL. In Netezza we usually call them UDF (user defined function), UDA (user defined aggregate), UDTF (user defined table function), or UDAP (user defined analytic process). The term UDX is often used to mean any type of user defined program. A UDX can only process data using the columns of data which are passed into the UDX, it has no capability to issue SQL to get additional lookup information etc. All of the logic must be based only on the arguments passed into the function.

Why SQL isn't allowed in a UDX

When SQL is used inside of a user defined function it limits the ability of the database to run the function in parallel. It is common in non parallel databases to allow functions to issue SQL statements, but in an massively parallel database that would limit scalability. In a MPP database a SQL statement is always executed across many nodes in parallel. Taking a TwinFin-12 as an example, when a SQL statement runs it is actually executing 92 parallel processes across 12 or 14 s-blades. If each of those parallel processes were to run a SELECT statement, each SQL statement would also run in parallel (creating 92 parallel processes for each statement), or a total of 92*92 = 8464 parallel processes. This of course wouldn't work as just one SQL statement would result in more processes than the database is capable of running. Databases which allow SQL to run inside of a function must therefore limit the functions from running in parallel. For that reason Netezza provides two distinct types of user defined programs. Stored procedures run as a single process on the host and are allowed to issue SQL. User defined functions run in parallel but cannot issue SQL.


Thursday, May 29, 2014

Netezza Basic Commands

Some Netezza basic commands use regularly

[nz@CELDNTZ1 ~]$ nzrev
Release 6.0.8 (P-11) [Build 32232]
[nz@CELDNTZ1 ~]$ nzstats

Field Name           Value
-------------------- ------------------------------
Name                 CELDNTZ1
Up Time              2941439 secs
Up Time Text         34 days, 1 hr, 3 mins, 59 secs
Date                 19-May-14, 17:50:38 SGT
State                8
State Text           Online
Model                IBM Netezza 100-1
Num SFIs             0
Num SPAs             1
Num SPUs             1
Num Data Slices      8
Num Hardware Issues  0
Num Dataslice Issues 0

[nz@CELDNTZ1 ~]$ nzhw -issues
No items found
[nz@CELDNTZ1 ~]$ nzds -issues
No items found
[nz@CELDNTZ1 ~]$ nzstate
System state is 'Online'.
[nz@CELDNTZ1 ~]$

Checking sessions (all/active)

nzsession -activeTxn

Netezza logs can be found in this directory

/nz/kit/log/postgres
/nz/kit/log/sysmgr
/nz/kit/log/startupsvr
/nz/kit/log/dbos

Sunday, May 25, 2014

Find unwanted tables in Production

As DBA, you will find your database does not just storing valid production tables but also other tables like TMP, BKP, BACKUP, DDMMYYYY format etc etc.  These tables take up space and especially during backup it will slow down process.  Below script will identify these tables:

SELECT the_table.database,
        CASE WHEN the_table.objclass = 4905  THEN 'Table    '
             WHEN the_table.objclass = 4910  THEN 'MView    '
             WHEN the_table.objclass = 4940  THEN 'TempTable'
             WHEN the_table.objclass = 4953  THEN 'Sec Table'
             WHEN the_table.objclass = 4951  THEN 'Sec Table'
             WHEN the_table.objclass = 4959  THEN 'Sec Table'
             WHEN the_table.objclass = 4961  THEN 'Table    '
             WHEN the_table.objclass = 4963  THEN 'Table    '
                                             ELSE 'What?    '
        END AS "Object",
        MAX
        (   CASE WHEN the_table.objclass = 4910                THEN   SUBSTR(the_table.objname, 3)::nvarchar(128)
                 WHEN the_table.objclass in (4951,4959,4963)   THEN NULL
                 WHEN the_table.objclass =  4940 then the_table.objname::nvarchar(128)
                        || nvl( '/' || _vt_tbldict.tblname,'')
                 ELSE   the_table.objname::nvarchar(128)
            END
        )
        ||
        MAX( case when the_table.objclass in (4959,4963) then ' (Versioned)' else '' end)
        AS "Name"
  ,       TO_CHAR ( nvl(SUM (  used_bytes  ),0)                     , '999,999,999,999,999'  ) AS "Bytes"
  ,          TO_CHAR ( nvl(SUM (  used_bytes  ),0) / (          1024 ) ,     '999,999,999,999'  ) AS "KB"
  ,          TO_CHAR ( nvl(SUM (  used_bytes  ),0) / (       1048576 ) ,         '999,999,999'  ) AS "MB"
  ,          TO_CHAR ( nvl(SUM (  used_bytes  ),0) / (    1073741824 ) ,             '999,999.0') AS "GB"
  ,          TO_CHAR ( nvl(SUM (  used_bytes  ),0) / ( 1099511627776 ) ,                 '999.0') AS "TB"
FROM  _V_OBJ_RELATION_XDB the_table
      left outer join _V_SYS_OBJECT_DSLICE_INFO on
      (
               the_table.objid = _V_SYS_OBJECT_DSLICE_INFO.tblid   and
               _V_SYS_OBJECT_DSLICE_INFO.tblid > 200000
      )
    left outer join _VT_TBLDICT on the_table.objid = _VT_TBLDICT.tblid
WHERE              the_table.objclass in (4905,4910,4940,4959,4951,4953,4961,4963)
               and the_table.objid > 200000
and
(
the_table.objname like '%0%'
or the_table.objname like '%1%'
or the_table.objname like '%2%'
or the_table.objname like '%3%'
or the_table.objname like '%4%'
or the_table.objname like '%5%'
or the_table.objname like '%6%'
or the_table.objname like '%7%'
or the_table.objname like '%8%'
or the_table.objname like '%9%')
and (the_table.objname not like 'NZCC_%'
and the_table.objname not like '_V_%'
and the_table.objname not like 'UAC_%'
and the_table.objname not like 'NZORCH_%'
and the_table.objname not like '%#%'
and the_table.objname not like '%$%')
GROUP BY the_table.DATABASE,
        "Object",
        the_table.visibleid;

Friday, May 23, 2014

Query to find database size

Need a query to find the database size in your Netezza?  Run the query below:

(
    SELECT                             'Database ' AS "Object",
                                       subselect.database::nvarchar(128) AS "Name"
  ,       TO_CHAR ( nvl(SUM ( used ),0)                     , '999,999,999,999,999'  ) AS "Bytes"
  ,          TO_CHAR ( nvl(SUM ( used ),0) / (          1024 ) ,     '999,999,999,999'  ) AS "KB"
  ,          TO_CHAR ( nvl(SUM ( used ),0) / (       1048576 ) ,         '999,999,999'  ) AS "MB"
  ,          TO_CHAR ( nvl(SUM ( used ),0) / (    1073741824 ) ,             '999,999.0') AS "GB"
  ,          TO_CHAR ( nvl(SUM ( used ),0) / ( 1099511627776 ) ,                 '999.0') AS "TB"
    FROM
        (
                SELECT                  database
                      ,0::bigint AS used
                          FROM        _v_database
                          WHERE       database not in ('master_db', 'MASTER_DB')
              UNION ALL
                SELECT                       the_table.database
                             ,SUM ( used_bytes ) as used
                FROM          _V_OBJ_RELATION_XDB the_table
                              left outer join _V_SYS_OBJECT_DSLICE_INFO on
                              (    the_table.objid = _V_SYS_OBJECT_DSLICE_INFO.tblid   and
                                   _V_SYS_OBJECT_DSLICE_INFO.tblid > 200000
                              )
                WHERE             the_table.objclass in (4905,4910,4940,4959,4951,4953,4961,4963)
                              and the_table.objid > 200000
                GROUP BY       the_table.visibleid
                              ,the_table.database
        ) subselect
                        GROUP BY  "Name"
)
UNION ALL
(
        SELECT
        'Appliance' AS "Object",
        'CELDNTZ1                        '::nvarchar(128) AS "Name"
  ,       TO_CHAR ( nvl(SUM ( used ),0)                     , '999,999,999,999,999'  ) AS "Bytes"
  ,          TO_CHAR ( nvl(SUM ( used ),0) / (          1024 ) ,     '999,999,999,999'  ) AS "KB"
  ,          TO_CHAR ( nvl(SUM ( used ),0) / (       1048576 ) ,         '999,999,999'  ) AS "MB"
  ,          TO_CHAR ( nvl(SUM ( used ),0) / (    1073741824 ) ,             '999,999.0') AS "GB"
  ,          TO_CHAR ( nvl(SUM ( used ),0) / ( 1099511627776 ) ,                 '999.0') AS "TB"
FROM
        (
                SELECT

                                      SUM ( used_bytes ) as used
                FROM          _V_OBJ_RELATION_XDB the_table
                              left outer join _V_SYS_OBJECT_DSLICE_INFO on
                              (    the_table.objid = _V_SYS_OBJECT_DSLICE_INFO.tblid   and
                                   _V_SYS_OBJECT_DSLICE_INFO.tblid > 200000
                              )
                WHERE             the_table.objclass in (4905,4910,4940,4959,4951,4953,4961,4963)
                              and the_table.objid > 200000
                GROUP BY      the_table.visibleid
        ) subselect
)
         order by 1 ASC, 2 ASC



Sample Output:

Object Name Bytes KB MB GB TB
Appliance CELDNTZ1                    6,014,002,790,400 5,873,049,600 5,735,400 5,601.0    5.5
Database CELEBI_STG_PROD_DB   1,535,959,891,968 1,499,960,832 1,464,806 1,430.5    1.4
Database AIU_HTTP_DB     840,035,467,264    820,347,136    801,120    782.3    .8
Database CELENR_HLD_PROD_DB     739,612,426,240    722,277,760    705,349    688.8    .7
Database CELEBI_NBA_DDS_DB     471,286,415,360    460,240,640    449,454    438.9    .4
Database SYSTEM     395,920,801,792    386,641,408    377,580    368.7    .4

Wednesday, May 21, 2014

query to find table in Netezza

If you need to search for table/tables in your databases, try query below.  Change the table name accordingly.  This is pretty much like Oracle DBA_TABLES/DBA_OBJECTS.

select database, objname, owner
from _v_obj_relation_xdb
where objname IN ('ARRANGEMENT_ACTIVITY','SUBSCRIBER_ARRANGEMENT')
and objtype='TABLE'

Tuesday, May 20, 2014

nz_migrate undocumented

#########################
# Undocumented Features #
#########################
#
# Normally, I would expect you to want to migrate ALL of a table (all rows).  But maybe
# you want to migrate just some of the data and/or break the migration up into smaller
# pieces.  Therefore, I allow you to specify optional SQL -- on a PER TABLE BASIS -- that
# can be used to control/limit the data extraction.
#
# You do this via environment variables.
# You can set them up for as many, or as few, tables as you wish.
# You set them on a per-table basis (one environment variable per table).
#
# If your table name was CUSTOMER_ORDERS, then you would set up this optional
# environment variable in the following manner, prior to launching nz_migrate.
#
#     export OPTIONAL_SQL_CUSTOMER_ORDERS="order_quantity > 1000"
#
# The environment variable must start with OPTIONAL_SQL_
# The tablename part is case sensitive.  It must match the source system exactly (e.g., upper or lower)
# You must set the variable to a valid SQL statement appropriate for this table.
# BASICALLY, this script is going to issue the first part of the SQL statement, e.g.
#           select * from customer_orders where
# and then append to it whatever you've specified
#           order_quantity > 1000
#
# No error checking will be performed.  Make sure you use valid SQL if you use this option.
# If this feature is used, it will be recorded in the nz_migrate output.  e.g.,
#
#      .....OPTIONAL SQL RESTRICTION                    order_quantity > 1000
#
# Any such restriction will also be passed to the nz_cksum portion of the script,
# so that it verifies+validates just the portion of the data that was moved.


# If your migration is going to take a long time ... if you want to migrate the
# data as of a particular point-in-time ... if the source tables might be changing,
# and you don't want those changes to affect the migration ... then you can add
# this switch
#                -stableTXid nnnn
# where nnnn is the stable transaction ID/number.  For more info (or to find out
# the current stableTXid) see:  nz_get_stableTXid -?


# By default, when larger tables are migrated, 4 threads are used.
#
# But what if ...
# say 3 of the threads finished, but one got hung up/failed for "some reason".
# Here is a way to redo just that one thread ... so you only need to (re)migrate
# that portion of the table.  (The less data you have to move over, the better
# off you'll be).
#
# a)  Terminate the nz_migrate job (and any jobs it may have left running on
#     either of the two boxes)
# b)  Did the target table have any rows inserted into it for this thread?
#     They'll get rolled back automatically.  But at some point in time you'll
#     want to nzreclaim that space.  A block level reclaim should suffice.
# c)  Lets say that "thread 2 of 4" failed.  So you want to redo just that thread.
#     Set the following environment variable
#            export MIGRATE_JUST_THIS_THREAD=2
# d)  Then (re)run nz_migrate against that database table ... using basically the
#     same options as before.  But now the script will just start up thread 2 (of 4)
#     rather than threads 1 thru 4.  It is all controlled by this environment
#     variable.