Enterprise UNIX

Book Notes: The World Is Flat (Part 2)

2007-11-25T19:44:00.000-05:00

Dealing with a Flat World. There is one main question I think is important to answer: "As the labor pool increases substantially with globalization, what are the most important skills to develop as an individual in order to remain employable & competitive?"

The book suggests that the best jobs (for those of us in middle-class America) will be in several fields as the world continues to flatten:

Great Collaborators and Orchestrators: "the manager who can work in and orchestrate 24/7/7 supply chains" (pg. 282). Somebody has to keep the work moving constantly moving through the system, across multiple continents (thus the 24/7/7- 24 hours a day, 7 days a week, across 7 continents)
The Great Synthesizers: people who can combine their expertise with another major discipline- areas that would not traditionally not go together, such as biology and computer science. "Search engine optimizers, for example, bring together mathematicians and marketing experts" (pg. 283).
The Great Explainers: With the new innovations that come from the Synthesizers, people will be needed who can understand the complexity of the innovation but explain it with simplicity.
The Great Leveragers: people who can see a problem, stop it, and then redesign the system so the problem never happens again. "It's all about combining the best of what computers can do with the best of what humans can do, and then constantly reintegrating the new best practices the humans are innovating back into the system to make the whole--the machines and the people--that much more productive" (pg. 289).
The Great Adapters: instead of being a generalist (broad skills but not deep) or a specialist (deep skills but not broad), a versatilist is someone who can "apply depth of skill to a progressively widening scope of situations and experiences, gaining new competencies, building relationships, and assuming new roles. Versatilists are capable not only of constantly adapting but also of constantly learning and growing" (pg. 289). These are people that have a Swiss Army Knife of skills to offer.
The Green People: simply stated, as countries continue to advance, the environment will continue to be a major issue. There will be plenty of jobs in addressing these challenges.
The Passionate Personalizers: Alan Blinder suggests that there will be a renewal of personally delivered services instead of impersonally delivered services by computer-generated voices or voices from India.
The Great Localizers: There is opportunity for small and medium-sized businesses to take advantage of the global capabilities that a flat world produces. "The localization of the global will be the freelancer who finds a way to use a satellite dish, a DSL line, a BlackBerry, a PC, or some new software to become a book editor or a film editor or an eBay entrepreneur from his or her bedroom" (pg. 295).

Other Notes:

CQ + PQ > IQ: Creativity + Passion > Intelligence.
"If you want to be sure that you are an untouchable", argues [Daniel Pink], a person with a job that "a computer or robot cannot do faster or some talented foreigner cannot do cheaper" and just as well, you need to focus on constantly developing your right-brain skills-"such as forging relationships rather than executing transactions, tackling novel challenges instead of solving routine problems, and synthesizing the big picture rather than analyzing a single component" (pg 307).
The American education system needs to be seriously revamped. There is not enough focus on science and engineering; there is not a passion in kids to learn- it is more of a chore. Instead of popularity being the result of intelligence, it is the result of wearing the right clothes and being involved in the right sports/activities. Our counterparts in China and India are extremely motivated to learn while we are moving at a snail's pace.

- John

Sarbanes Oxley and UNIX Privileged Access

2007-11-21T13:35:00.000-05:00

Sarbanes-Oxley section 404 is summarized as follows:

Issuers are required to publish information in their annual reports concerning the scope and adequacy of the internal control structure and procedures for financial reporting. This statement shall also assess the effectiveness of such internal controls and procedures. The registered accounting firm shall, in the same report, attest to and report on the assessment on the effectiveness of the internal control structure and procedures for financial reporting.

How this translates into UNIX privileged access is a topic of much confusion. After researching, there are several decent resources out there that help to clarify:

From a white paper, Controlling Privileged Accounts by Fox Technologies (manufacturer of Keon/BoKS):

For a SOX audit, it is no longer sufficient to say you trust your administrators; you must have controls in place to convince your auditors that no administrator, trustworthy or not, is able to abuse the authority granted.

Organizations struggling to resolve these issues often end up evaluating three different alternatives:

1. Create home-grown solutions based on Operating System capabilities, available utilities such as “sudo”, clever password management procedures, and lots of scripts. Except for in very small organizations, these attempts will either become extremely costly with system administrators programming instead of doing their jobs. This approach is often found insufficient from an auditor’s perspective. Even if home-grown solutions achieve an acceptable level with regard to password management, they fail to provide corresponding auditing capabilities.

2. Combine various commercial or open source point-solutions, to create an operating environment that takes many of these requirements into account. This typically involves using one solution for user provisioning, another for centrally managed secure communications (SSH), a third for password management or other types of root account management, a fourth for keystroke logging, and yet another tool for audit log consolidation. This could actually amount to something quite powerful in the end, yet one important aspect by necessity is lost: centralized management on one security system. Combining multiple technical solutions into one leaves conceptual gaps which in turn leads to security flaws and inefficient management. All things considered, this is not a cost-efficient approach although, ironically, cost-awareness may well be the primary driver for organizations exploring this option.

3. Invest in an Enterprise Access Management solution (EAM). These solutions are everything but lightweight and in reality there are only a couple of vendors offering full-blown EAM solutions. This third alternative is actually what analysts Jay Heiser and Ant Allan have recommended for larger organizations. “Gartner advocates use of an enterprise access management product for large and complex organizations that can derive benefit from having an external access control system for multiple Unix targets. Although these products do address the control of superuser privileges, they do much more beside and, consequently, are more expensive and more complex to install than the Unix-focused tools.” (Controlling Unix Superuser Privileges Is Critical, Gartner Research G00130427, August 31, 2005).

It is not difficult to see the rationale for the Gartner recommendation: Alternative 1 and 2 do not provide a complete solution. Yet, apart from the fact that EAM packages are “more expensive and more complex to install” and to operate, they may also fail to deliver all necessary components.

From a different paper, Delegating Root Authority and Auditing Activities on UNIX/Linux Systems:

Most native operating systems in the UNIX and Linux world, generally regardless of vendor, fail to meet the required levels of accountability required for Sarbanes-Oxley compliance, though SELinux goes some way toward correcting these deficiencies. The simplest administrative tasks require users to have access to the root account, which has no granularity of control in the native environment, leading to an abstract picture of which users have had access to and have modified data.
...
Section 404 is especially important to IT managers, because
companies must have begun to comply by 15 November 2004,
and must be able to verify the following for their CEOs and
CFOs to sign off on their annual assessment:
• Access controls surrounding financial data
• Data encryption
• Authorization to access and modify systems
• Systemwide intrusion monitoring
• Intrusion response
• Indelible auditing

There's another paper out there that discusses Sarbanes Oxley and UNIX at large: Unix and Sarbanes-Oxley: a management and auditors guide. It doesn't go into much depth on the specific issues addressed above, but discusses some items .

Symark (manufacturer of PowerKeeper) has several papers that seem to be potentially interesting (free registration required), although some of it seems to be more marketing-focused:

Meeting the Access Security Requirements of Sec 404 of the Sarbanes-Oxley Act in a Heterogeneous UNIX/Linux Environment
Guide to Creating a Secure Access Control Environment
Passing UNIX/Linux Audits and Meeting Regulatory Compliance

On a related note, Gartner has published some good research on this topic with their paper Toolkit: Password Management Tools for Shared Accounts and Service Accounts. This paper is not free; I believe it is available for purchase.

- John

Book Notes: The World Is Flat (Part 1)

2007-11-20T22:02:00.000-05:00

I've been reading Thomas Friedman's book, The World Is Flat. It's been a real eye-opener. I'm planning on breaking this review / commentary into 2 parts: 1 on the "flatteners" and 1 on how individuals can deal with the flat world.

Here are the flatteners:
#1: 11/9/89: The New Age of Creativity: When the Walls Came Down and the Windows Went Up. This is the fall of the Berlin Wall. The flattening effect was the domino effect that ended up resulting in the world embracing capitalism.

#2: 8/9/95: The New Age of Connectivity: When the Web Went Around and Netscape Went Public. The invention of the Internet and the Web set the framework and the foundation for all of us to be able to interact across the planet in an instantaneous and graceful manner.

#3: Work Flow Software: In essence, the standardization of the various formats and platforms in which we communicate over the Internet allow us to be more efficient and more effectivel. I can send you a Microsoft Word document, and you can send me an Excel Spreadsheet. We can both look at the same JPGs or work on the same software through a CVS or Subversion repository. AJAX, HTML, & XML are all in this category as well.

#4: Uploading: The vast majority of humanity has become accustomed to being the recipients of information. We read books, magazines, learn through courses, watch TV, and listen to music. The development of the web allows us to all be publishers as well. I can start a Blog (just like this one!) that the world can see. Online communities form to allow people from across the globe to share their thoughts, feelings, and expertise on a seemingly infinite range of topics and projects. The development of the Apache web server through the Open Source Software movement is a prime example of this type of community. What's more, with Open Source software, everything that goes into a full release has been peer reviewed by some of the best and brightest developers in the community. Wikipedia is another famous project that demonstrates the power of Uploading.

#5: Outsourcing: Y2K. When Y2K troubled the world, there was a clear need for a cheap and effective fix. The tech bubble had spawned the laying of lots of transcontinental fiber optic lines, and this created very fast connectivity capabilities internationally. The result was that Indian (particularly those educated through India's Institutes of Technology (IIT)) engineers came to the rescue. American companies learned to work with India's engineers; they learned that practically anything that could be digitized could be outsourced at a fraction of the cost.

#6: Offshoring: Running with Gazelles, Eating with Lions. In other words, China. China continues to reduce the cost of manufactured goods all over the world. It is important to note that the book proposes that China is not a "lose-lose" situation for American workers. From pg. 146 of the book:

According to the US Commerce Department, nearly 90 percent of the output from US-owned offshore factories is sold to foreign consumers. [This] actually stimulates American exports. There is a variety of studies indicating that every dollar a company invests overseas in an offshore factory yields additional exports for its home country, because roughly one-third of global trade today is within multinational companies.

#7: Supply-Chaining: Eating Sushi in Arkansas. Wal-Mart's supply chain is the model for this flattener. While Wal-Mart is the largest retail company in the world, it does not actually make anything- it just sells other peoples products in a remarkably efficient way. Companies must take advantage of the best producers at the lowest price (to include manufacturing as well as transportation costs) from anywhere in the world. Technology allows companies to properly forecast demand and match it with supply (ex: RFID).

#8: Insourcing: What the Guys in Funny Brows Shorts Are Really Doing. The example here is UPS, but not what I'm used to. UPS has become a company that can reach into your company's business process and take care of the logistics, allowing you to focus on what your expertise is. The example that I thought was particularly insightful was the story of Toshiba laptops- when a laptop breaks, UPS picks it up, sends it to its Louisville hub, and UPS employees who are trained by by Toshiba fix it. This is how we can have a repaired computer back to us in 3 days (1 day to get it out, 1 day to fix it, 1 day to get it back). This flattener allows small companies to act with large presence, and large companies to have their complex supply chains managed for them. In some instances, companies don't even really do much other than innovate. When an order is placed for Nike shoes on their website, UPS handles the entire transaction, from picking out the shoes in a warehouse to delivering them to your door. UPS can even take care of the billing. Wow. What an eye-opener.

#9: In-Forming. Google, Yahoo!, & MSN Web Search. This one is obvious to me. With the maturation and development of the killer-app "Search", we have practically limitless information at our fingertips. Important too is to "be good"- in today's world, your name is more and more likely to come up on searches, so anything bad you do could turn up.

#10: The Steroids. Digital, Mobile, Personal, and Virtual. All of the flatteners seem to build on each other, and these new(er) technologies are the icing on the cake. By digitizing things, we can plug them into the world (ex: look at how digital cameras have changed photography and how MP3s are changing music). By making things mobile, we can tap into the world anywhere (ex: wireless internet connectivity and cell phones). By personalizing, our preferences and tastes are known (ex: Amazon book & Netflix movie recommendations). With virtualization, location becomes even less important (ex: VOIP & VPNs).

- John

AIX Links

2007-11-19T13:10:00.000-05:00

Here's some links to some other Blogs / Sites that I've found helpful for AIX related support:

P-Series Support Forum: http://www.pseriestech.org/forum/
IBM AIX Wiki: http://www-941.ibm.com/collaboration/wiki/display/WikiPtype/Home
IBM AIX Redbooks: http://www.redbooks.ibm.com/redbooks.nsf/Portals/UNIXTop10

- John

Veritas Netbackup: Part 5- Logs Directory

2007-11-19T12:22:00.000-05:00

This directory (/usr/openv/netbackup/logs) contains the user_ops sub-directory and, optionally, sub-directories where detailed debug logs will be created.

The directory /usr/openv/logs contains the sub-directories where the detailed debug logs for the vnetd and vopied daemons will be created. You will need to create the /usr/openv/logs directory if it does not exist and you want debug logs for vnetd or vopied.

The user_ops sub-directory is created during the install of NetBackup on all servers and clients. It is used by the NetBackup - Java GUI applications for placement of job and progress log files generated by the end-user client application (jbp) and temporary files generated while using these applications. This directory must exist for successful operation of any of these GUI applications and must have public read, write and execute permissions. One sub-directory in logs/user_ops will exist for every user that is using these GUI applications. The pruning of files in this sub-directory is done per the semantics of the KEEP_LOGS_DAYS bp.conf option - default is 3 days.

All other sub-directories in /usr/openv/netbackup/logs are optional and should only be created if unexplained problems are occurring with the NetBackup product and more information is required to isolate the problem.

Miscellaneous notes about these optional NetBackup logs contained in this directory:

Each NetBackup process has its own debug log directory
One debug log file is created per process per day. The file names created are of the form: log. (e.g., log.110891)
Debug logging is only in affect for a process if that process's debug log directory is defined.
The log files in these directories are automatically deleted by the NetBackup request daemon, bprd. The administrative parameter "keep logs x days" determines how long the log files exist.
Volume of debug logging can be increased by enabling "verbose" mode by defining the string "VERBOSE" in the /usr/openv/netbackup/bp.conf file. WARNING: Some of these logs can potentially grow very large, and should only be enabled if unexplained problems exist.

The following directories can be individually created on the server, to cause the corresponding NetBackup server process to log information:

/usr/openv/netbackup/logs/admin
/usr/openv/netbackup/logs/bpbrm
/usr/openv/netbackup/logs/bpbrmds
/usr/openv/netbackup/logs/bpcd
/usr/openv/netbackup/logs/bpcoord
/usr/openv/netbackup/logs/bpdbm
/usr/openv/netbackup/logs/bpdm
/usr/openv/netbackup/logs/bpjava-msvc
/usr/openv/netbackup/logs/bpjava-susvc
/usr/openv/netbackup/logs/bpjobd
/usr/openv/netbackup/logs/bprd
/usr/openv/netbackup/logs/bpsched
/usr/openv/netbackup/logs/bpsynth
/usr/openv/netbackup/logs/bptm
/usr/openv/netbackup/logs/symlogs
/usr/openv/logs/vnetd
/usr/openv/logs/vopied

In order for non-root administrators to be able to write to the log files,
the "admin" directory should be created with access modes of 777.

The following directories can be individually created on the client, to cause
the corresponding NetBackup client process to log information:

/usr/openv/netbackup/logs/bp
/usr/openv/netbackup/logs/bparchive
/usr/openv/netbackup/logs/bpbackup
/usr/openv/netbackup/logs/bpbkar
/usr/openv/netbackup/logs/bpcd
/usr/openv/netbackup/logs/bpdbsbora
/usr/openv/netbackup/logs/bpfilter
/usr/openv/netbackup/logs/bpjava-msvc
/usr/openv/netbackup/logs/bpjava-usvc
/usr/openv/netbackup/logs/bpkeyutil
/usr/openv/netbackup/logs/bplist
/usr/openv/netbackup/logs/bpnbat
/usr/openv/netbackup/logs/bporaexp
/usr/openv/netbackup/logs/bporaexp64
/usr/openv/netbackup/logs/bporaimp
/usr/openv/netbackup/logs/bporaimp64
/usr/openv/netbackup/logs/bprestore
/usr/openv/netbackup/logs/bphdb
/usr/openv/netbackup/logs/dbclient
/usr/openv/netbackup/logs/symlogs
/usr/openv/netbackup/logs/tar
/usr/openv/logs/vnetd

In order for users on the client to be able to write to the log files, the directories should be created with access modes of 777.

Here are descriptions of NetBackup processes:

bprd

request daemon
can be terminated and initiated from the admin interfaces
responds to client and administrative requests
restores
backups
archives
"list files backed-up or archived"
manual/immediate backups
reread configuration database

bpsched

backup scheduler
started by bprd on user directed backups and archives
started by bprd on immediate/manual backups
started by bprd every "Wakeup Interval" for regularly scheduled incremental and full backups
uses information from the policy & storage unit databases to determine what clients to start, when to start them, and what storage unit to write backups/archives to

bpdm

disk manager
used on storage units of type Disk
started by bpbrm on backups and restores
during backups and restores, one of these is started (on the server with the storage unit) for each client backup or restore

bptm

removable media (tape) manager
used on storage units of type Logical Tape
started by bpbrm on backups and restores
during backups and restores, one of these is started (on the server with the storage unit) for each client backup or restore
also responsible for managing the media database
used to display info in the Media Reports screen when you select Media List

bpbrm

backup/restore manager
started by bpsched on backups/archives
started by bprd on restores
during backups and restores, one of these is started (on the server with the storage unit) for each client backup or restore
responsible for managing both the client and the media manager processes. uses error status from both to determine ultimate status of backup or restore.

bpbrmds

backup/restore manager for Disk Staging duplications
started by bpsched when handling a Disk Staging schedule
responsible for starting, and managing, a bpduplicate process.

bpdbm

database manager
manages policy, config/behavior, storage unit, and error DB's

bpjobd

job manager
manages backup and restore jobs for the activitiy monitor GUIs

bpsynth

NetBackup synthetic backup manager
started by bpsched on synthetic backups
runs on master server and manages the creation of a synthetic image.

bpcoord

NetBackup synthetic backup read coordinator
started by bpsynth for synthetic backups
runs on master server and coordinates reading required blocks from existing images.

bpcd

"client daemon"
used on clients (and remote servers) to initiate other product programs, without requiring /.rhosts entries for the server on each client
started by the inetd(1M) process

bpjava-msvc

NetBackup-Java application server authentication service program
started by inetd during startup of the NetBackup-Java GUI applications
authenticates the user that started the NetBackup-Java GUI application

bpjava-susvc

NetBackup-Java application server user service program on NetBackup servers
Started by bpjava-msvc upon successful login via the NetBackup-Java GUI applications login dialog window
services all requests from the NetBackup-Java GUI applications for administration and end-user operations on the host on which the NetBackup-Java application server is running
additional bpjava-susvc processes get started to respond to requests from the NetBackup-Java GUI applications

bpjava-usvc

NetBackup-Java application server user service program on NetBackup clients
started by bpjava-msvc upon successful login via the NetBackup-Java GUI applications login dialog window
services all requests from the NetBackup-Java GUI applications for administration and end-user operations on the host on which the NetBackup-Java application server is running
additional bpjava-usvc processes get started to respond to requests from the NetBackup-Java GUI applications

bpdbjobs: command-line activity monitor for backup and restore jobs

bparchive

command-line program on clients to initiate archives
communicates with bprd on server

bpbackup

command-line program on clients to initiate backups
communicates with bprd on server

bpbkar

program used on standard clients to generate backup images
not used directly by client users

bplist

command-line program on clients to initiate file lists
communicates with bprd on server

bprestore

command-line program on clients to initiate restores
communicates with bprd on server

bpnbat: command-line program to manage NetBackup Access Control authentication
bphdb: program used to start obackup to do Oracle database backups
dbclient: log for the DB Extension clients
tar: program used on standard clients to restore backup images
wbak: program used on Apollo clients to generate backup images
rbak: program used on Apollo clients to restore backup images
bp: menu user interface for backups, archives, and restores
xbp: X/Motif user interface for backups, archives, and restores
bpadm: menu user interface for administrative tasks

bpdbsbora

command-line program used to initiate template based user-directed backup and recovery
enables generation of a shell script from a template

bporaexp

command-line program on clients to export Oracle data in XML format
communicates with bprd on server

bporaexp64

64-bit command-line program on clients to export Oracle data in XML format
communicates with bprd on server

bporaimp

command-line program on clients to import Oracle data in XML format
communicates with bprd on server

bporaimp64

64-bit command-line program on clients to import Oracle data in XML format
communicates with bprd on server

bpfilter: client program used to filter backup images between the media server and bpbkar or tar on the client

bpkeyutil: command-line program to manage Encryption key files

vnetd

VERITAS Network Daemon
Used to create "firewall friendly" socket connections
started by the inetd(1M) process

vopied

VERITAS OPIE authentication Daemon
Used by servers to authenticate non-root users via VOPIE authentication
started by the inetd(1M) process

For lots more discussion and information on Veritas Netbackup, see the Symantec forum over at https://forums.symantec.com/syment/board?board.id=21

Veritas Netbackup: Part 4- Exclude Lists

2007-11-19T12:17:00.000-05:00

Creating an Exclude List on a UNIX Client
If you create a /usr/openv/netbackup/exclude_list file on a UNIX client, NetBackup uses the contents of the file as a list of patterns to skip during automatic full and incremental backups.

Note Exclude and include lists do not apply to user backups and archives.

The following types of files typically appear in an exclude list:

*.o files
core files
a.out files
Files prefixed or suffixed by ~ (backups for editors)
Files and directories under /tmp, /usr/tmp
Man pages
Software that you can restore from original installation tapes
Automounted directories
CD-ROM file systems
NetBackup automatically excludes the following file system types:
mntfs (Solaris)
proc (all UNIX platforms)
cdrom (all UNIX platforms)
cachefs (AIX, Solaris, SGI, UnixWare)

Note VERITAS suggests that you always specify automounted directories and CD-ROM file systems in the exclude list. Otherwise, if they are not mounted at the time of a backup, NetBackup must wait for a timeout before proceeding.

Syntax Rules
The following syntax rules apply to exclude lists:

Blank lines or lines beginning with a pound sign (#) are ignored.
Only one pattern per line is allowed.
The following special or wildcard characters are recognized: [ ] ? * { }

To use special or wildcard characters literally (that is, as non-wildcard characters), precede them with a backslash (\). For example, assume the brackets in the following are to be used literally
/home/abc/fun[ny]name

In the exclude list, precede them with a backslash as in

/home/abc/fun\[ny\]name

Note A backslash (\) acts as an escape character only when it precedes a special or wildcard character as in the above example. This means that NetBackup normally interprets a backslash literally and it is a legal character to use in pathnames.

If you exclude all files in the backup selections list by using / or * or both symbols together (/*), NetBackup backs up only what is specified by full path names in the include list.
Spaces are considered legal characters. Do not include extra spaces unless they are part of the file name.

For example, if you want to exclude a file named
/home/testfile (with no extra space character at the end)
and your exclude list entry is
/home/testfile (with an extra space character at the end)
NetBackup cannot find the file until you delete the extra space from the end of the file name.

End a file path with / to exclude only directories with that path name (for example, /home/test/). If the pattern does not end in / (for example, /usr/test), NetBackup excludes both files and directories with that path name.
To exclude all files with a given name, regardless of their directory path, just enter the name without a preceding slash. For example: "test" rather than "/test" This is equivalent to prefixing the file pattern with /, /*/, /*/*/, /*/*/*/ and so on.
Do not use patterns with links in the names. For example, assume /home is a link to /usr/home and /home/doc is in the exclude list. The file is still backed up in this case because the actual directory path, /usr/home/doc, does not match the exclude list entry, /home/doc.

Example of an Exclude List
In this example, an exclude list contains the following entries:


# this is a comment line
/home/doe/john
/home/doe/abc/
/home/*/test
/*/temp
core

Given the exclude list above, the following files and directories are excluded from automatic backups:

The file or directory named /home/doe/john.
The directory /home/doe/abc (because the exclude entry ends with /).
All files or directories named test that are two levels below home.
All files or directories named temp that are two levels below the root directory.
All files or directories named core at any level.

Exclude Lists for Specific Policies or Schedules

NetBackup allows you to create an exclude list for a specific policy or a policy and schedule combination. To do this, create an exclude_list file with a .policyname or .policyname.schedulename suffix. The following are two examples for a policy named wkstations that contains a schedule named fulls:

/usr/openv/netbackup/exclude_list.wkstations
/usr/openv/netbackup/exclude_list.wkstations.fulls

The first file affects all scheduled backups in the policy named wkstations. The second file affects backups only when the schedule is named fulls.

For a given backup, NetBackup uses a single exclude list—the list containing the most specific name. For example, if there are files named:

exclude_list.wkstations and exclude_list.wkstations.fulls

NetBackup uses only:

exclude_list.wkstations.fulls

Veritas Netbackup: Part 3- UNIX Raw Partitions

2007-11-19T12:14:00.000-05:00

Caution Save a copy of the partition table before performing raw-partition backups so you have it for reference prior to a restore. To restore the raw partition, a device file must exist and the partition must be the same size as when it was backed up. Otherwise, the results of the restore are unpredictable.

Notes On UNIX Raw-Partition Backups

Use raw-partition backups only if you can ensure that the files are not changed in any way during the backup or, in the case of a database, if you can restore the database to a consistent state by using transaction log files.
Do not perform archives of raw partitions on any client. An archive backs up the raw partition and then deletes the device file associated with the raw partition. However, the file system does not recover the space used by the raw partition.
Before backing up file systems as raw partitions, unmount the file system to allow buffered changes to be written to the disk, and to prevent the possibility of the file system changing during the backup. You can use the bpstart_notify and the bpend_notify scripts to unmount and remount the backed-up file systems.
The Cross Mount Points attribute has no effect on raw partitions. If the root partition is being backed up as a raw partition and has mount points for other file systems, the other file systems are not backed up, even if you select Cross Mount Points.

The same is true for the Follow NFS attribute. NFS file systems mounted in a raw partition are not backed up. Nor can you back up raw partitions from other machines by using NFS mounts to access the raw partitions. The devices are not accessible on other machines through NFS.

For disks managed by disk volume managers such as VERITAS Volume Manager (VxVm), specify the logical partition names.
For clients in a FlashBackup policy, refer to the NetBackup Advanced Client System Administrator’s Guide (backup selection list and cache section) for the differences between Standard and FlashBackup policies.

When to Use Raw-Partition Backups
If there are no file systems to back up and the disks are used in raw mode (such as with some databases), back up the disk partitions as raw partitions. When backing up databases as raw partitions, you can use the bpstart_notify and bpend_notify scripts to do the preprocessing and postprocessing necessary to back up the databases.

You can also perform a raw-partition backup of a disk partition used for file systems. A disadvantage of this method is that you must restore the entire partition to recover a single file (unless you are using FlashBackup). To avoid overwriting the entire partition, use the redirected restore feature to restore the raw partition to another raw partition of the same size, and then copy individual files to the original file system.

Raw-partition backups are also useful for backing up entire disks. Since the overhead of the file system is bypassed, a raw-partition backup is usually faster. The size of the raw-partition backup will be the size of the entire disk, regardless of whether the entire disk is used.

To specify a UNIX raw partition in the policy backup selection list, enter the full path name of the device file. For example, on Solaris:
/devices/sbus@1,f8000000/esp@0,800000/sd@2,0:1h

Caution Do not specify wildcards (such as /dev/rsd*) in paths for raw-partition backups. Doing so can prevent the successful restore of entire devices, if there is overlap between the memory partitions for different device files.

You can include raw partitions in the same backup selection list as other backups. For example:
/home
/usr
/etc
/devices/sbus@1,f8000000/esp@0,800000/sd@2,0:1h

Note NetBackup does not distinguish between full and incremental backups when backing up a raw partition. The entire partition is backed up in both cases.

Raw-partition backups occur only if the absolute file path in the backup selection list is a block or character special-device file. You can specify either block or character special-device files; although, character special-device files are often faster because character devices avoid the use of the buffer cache for accessed disk data. To obtain the optimum backup speed for raw-partition backups, test both a block and character special-device file to ensure the best choice for your platform.

Ensure that you are specifying the actual block-or character-device files. Sometimes, these are links to the actual device files. If a link is specified, only the link is backed up. If the device files are reached while backing up /dev, NetBackup backs up only the inode files for the device, not the device itself.

Selecting a Schedule Backup Type for a UNIX Raw Partition
When performing a raw partition backup, be sure to select Full Backup for the Type of Backup from the Schedules tab. Any other backup type will not work for backing up raw partitions. (See “Type of Backup” on page 146 of the Netbackup Guide.)

Veritas Netbackup: Part 2- Selection Lists & NFS

2007-11-19T12:11:00.000-05:00

NFS mount points have rules associated with them based on the client's configuration:

File-Path Rules for UNIX Clients
The general requirements for pathnames on UNIX clients are as follows:

Enter one pathname per line. NetBackup supports a maximum path length of 1023 characters on UNIX clients.
Start all pathnames with a slash (/).
You can use the following meta or wildcard characters in policy backup selection lists: *, ?, [ ], { }

The following are example UNIX file specifications that use this capability:

/home/.[a-zA-Z0-9]*
/etc/*.conf

To use meta or wildcard characters literally, precede them with a backslash (\). Assume, for example, that the brackets in the following pathname are used as literal characters: /home/abc/fun[ny]name

In the backup selection list, precede the brackets with a backslash as in
/home/abc/fun\[ny\]name

Note A backslash (\) acts as an escape character only if it precedes a meta or wildcard character. NetBackup normally interprets a backslash literally and it is a legal character to use in pathnames.

Notes on Backup Selection Lists for UNIX Clients

File paths that cross mount points or that the client mounts through NFS can affect the way that you must configure your backups. Before creating a backup selection list, familiarize yourself with the Follow NFS and Cross mount points attributes.
You can back up operating system, kernel, and boot files with NetBackup. You cannot, however, create bootable tapes. Consult your system documentation to create a bootable tape.
NetBackup never backs up the following:

NFS files or directories, unless you set Follow NFS.
Files or directories in a different file system if you do not set Cross mount points.
Files or directories with path lengths longer than 1023 characters.
Files or directories where the operating system does not return inode information (the lstat system call failed).
Directories that NetBackup cannot cd into.
On a disk managed by Storage Migrator, migrated files or directories where Storage Migrator does not return inode information (mig_stat fails). Note that NetBackup Server does not support Storage Migrator.
Socket special files (named pipes are backed up).
Locked files when mandatory locking is enabled by an application that currently has the file open.
Busy files. If a file is open, NetBackup backs up the last saved version of the file.

Exclude specific files from backups by creating an exclusion list on the client.
The BUSY_FILE_ACTION and LOCKED_FILE_ACTION options in the /usr/openv/netbackup/bp.conf file on the client offer alternatives for handling busy and locked files. See “NetBackup Configuration Options” on page 134 of the Netbackup Guide.
On Hewlett-Packard, AIX, Sequent, and Solaris 2.5 (and later) platforms, NetBackup backs up access control lists (ACLs).
NetBackup can back up (and restore) Sun PC NetLink files.
On IRIX 6.x and Digital Alpha platforms, NetBackup backs up extended file attributes.
On IRIX platforms, NetBackup backs up and restores extended attributes attached to XFS file system objects.
On DEC OSF/1 platforms, NetBackup backs up and restores extended attributes attached to files on AdvFS and UFS file systems.
By default, NetBackup backs up and restores Solaris 9 extended attribute files. The FlashBackup single file restore program (sfr) does not restore extended attribute files. (See “Backup and Restore of Extended Attribute Files and Named Data Streams” on page 119 of the Netbackup Guide.)
By default, NetBackup backs up and restores VxFS 4.0 named data streams. The FlashBackup single file restore program (sfr) does not restore extended attribute files. (See “Backup and Restore of Extended Attribute Files and Named Data Streams” on page 119 of the Netbackup Guide.)
On Hewlett-Packard and Solaris 2.5 (and later) platforms, NetBackup backs up VxFs extent attributes.
If there are one or more trailing spaces in a backup selection list entry and a matching entry is not found on the client, NetBackup deletes trailing spaces and checks again. If a match is still not found, NetBackup skips the entry and logs a message similar to one of the following in the NetBackup All Log Entries or Problems report:

TRV - cannot process path pathname: No such file or directory. Skipping
TRV - Found no matching file system for pathname

Symbolic Links to Files or Directories

For symbolic (soft) links, include the file path to the source file in your list in order to back up the actual data. If a file is a symbolic link to another file, NetBackup backs up only the link, not the file to which the link points. This prevents multiple backups of the source file.
Because symbolic links are restored only as a symbolic link to the source file, you must restore the source file along with the link in order to get the data.

Note: If NetBackup restores a symbolic link as root, it changes the owner and group back to the original owner and group. When NetBackup restores a UNIX symbolic link as a nonroot user, it sets the owner and group for symbolic links to the owner and group of the person doing the restore. This does not cause problems because when the UNIX system checks permissions, it uses the owner and group of the file to which the symbolic link points.

Hard Links to Directories
On most UNIX systems, only the root user can create a hard link to a directory. Some systems do not permit hard links and many vendors warn you to avoid using these links.
NetBackup does not back up and restore hard-linked directories in the same manner as it does files:
· During a backup, if NetBackup encounters hard-linked directories, it backs them up multiple times, once for each hard link.
· During a restore, NetBackup restores multiple copies of the hard-linked directory contents if the directories do not already exist on the disk. If the directories exist on disk, NetBackup restores the contents multiple times to the same disk location.

Hard Links to Files
A hard link differs from a symbolic link in that it is not a pointer to another file, but is actually two directory entries pointing to the same inode number.

During a backup, if the backup selection list includes hard-linked files, the data is backed up only once, using the first file name reference found in the directory structure. If a second or subsequent file name reference is found, it is backed up as a link to the name of the first file. This means you get only one backup copy of the data, regardless of whether you include one or multiple hard links. You can include any of the paths that are hard links to the data in order to back up the data.

During a restore, if all of the hard-link references are restored, the hard-linked files still point to the same inode as the other files to which they are linked. However, if you do not restore all the hard links, you can encounter anomalies as shown in the following examples.

Veritas Netbackup: Part 1- Introduction

2007-11-19T11:52:00.001-05:00

Veritas Netbackup is generally a very solid product in my experience, but sometimes in-depth troubleshooting is necessary. It is important to first get definitions straight for semantical purposes:

A Netbackup client is the computer/server that is backing up its files/directories/volumes/etc.
A Netbackup media server is the destination computer that has the tape library (or equivalent) attached to it. The client's data is physically written to tape through the media server.
A Netbackup master server is the management arm of Netbackup- it keeps track of the backups, archives, restores, policies, etc.

NFS Troubleshooting

2007-11-19T11:33:00.000-05:00

While in theory, NFS is supposed to be a great technology, my experience with it has been that it works well when the demand on it is not great. I don't like it for mission-critical purposes, in implementations that have long-standing NFS mounts, or those that have high throughput.

Too often NFS gets "stuck" and really messes up a server. Despite soft mounts claiming to take this away, we have seen time and time again NFS cause us pain. Df commands don't return properly, and the system behaves very strangely.

Trying to connect a UNIX server to the Microsoft version of NFS - "UNIX Services for Windows" is an utter nightmare.

So rule #1: don't use NFS if you don't have to. I'll take FTP transactions over NFS any day.

If you do get into NFS issues, here are a few commands I've kept in my archives on troubleshooting NFS (particularly on Solaris servers):

Run "ps -ef | grep nfsd" a few times & see if cpu time is accumulating
Run "ps -ef | grep mountd". If it is not running maybe run /usr/lib/nfs/mountd
From a NFS client, run "showmount -e server" or "dfshares server" to see what shares look like from the client perspective
From the client: "/usr/sbin/rpcinfo -u server nfs". This should give info on what versions of NFS are configured and waiting for connections. Also run "/usr/sbin/rpcinfo -u server mountd" for details
Check "/etc/dfs/dfstab" (or equivalent such as "/etc/exports") to make sure they are configured properly

Last resort- reboot the server; sometimes this can clean things up.

- John

Korn Shell (ksh) flow control

2007-11-09T10:59:00.000-05:00

Decision making ability, either on the command line or in a script allows for you to add intelligence to your commands. It is very important to check and double check your logic; getting the syntax correct is very important.

IF...THEN...ELSE (use ELIF for "Else If")


(( _num1_ == _num2_ ))           numbers equal
(( _num1_ != _num2_ ))           numbers not equal
(( _num1_ <> _num2_ ))            num1 > num2
(( _num1_ <= _num2_ ))           num1 <= num2 (( _num1_ >= _num2_ ))           num1 >= num2
[[ _str1_ == _str2_ ]]           strings equal
[[ _str1_ != _str2_ ]]           strings not equal
[[ _str1_ <> _str2_ ]]            str1 follow str2
[[ _str1_ = _pattern_ ]]         str1 = pattern
[[ _str1_ != _pattern_ ]]        str1 != pattern
[[ -z _str_ ]]                   str is null
[[ -n _str_ ]]                   str is not null

You can also use the following when examining files:


object exist                    -a
readable                        -r
writable                        -w
executable                      -x
non-zero length                 -s
zero length (null)              -z
directory                       -d
plain file                      -f
symbolic link                   -h
named pipe                      -p
block special file              -b
character special file          -c
soft link                       -L
socket                          -S

Flow control statements can take 2 forms:
Block Form:

if (( `cat /etc/passwd|grep John|wc -l` > 0 )); then
  echo 'John exists'
else
  echo "John doesn't exist"
fi

Line Format- semicolon placement. Semicolons only come after "standard" shell statements, not flow control statements. Pay careful attention to the placement in this example:
(note: I'm using a "\" character to break up lines that are too long; you normally wouldn't do this)

if ((`cat /etc/passwd|grep John|wc -l`>0)); then \
 echo "John exists"; else echo "no John"; fi

Another Example: be sure to use a space after opening and closing square brackets
(note: I'm using a "\" character to break up lines that are too long; you normally wouldn't do this)

if [[ -f /etc/passwd ]]; then \
echo "/etc/passwd is a regular file!"; fi

if [[ -f /etc/passwd && -r /etc/passwd ]]; then \
echo "/etc/passwd is a file and is readable"; fi

Parsing Text

2007-11-09T10:08:00.001-05:00

One of powerful benefits of UNIX related operating systems (as opposed to Windows) is the built-in powerful text parsing commands and interfaces. We rely on these to combine, sort, and extract text from all sorts of sources. Here are a few of my favorites:

Grep: most useful for extracting lines out of a command or file
Examples:

cat /etc/passwd | grep John
This method uses "standard in" as the file being used

grep John /etc/passwd
This method uses a real file for processing

Frequently Used Options:
-i: case insensitive
-v: "reverse" grep: print everything except a matching line
-n: print line numbers

EGrep: "Extended" Grep

cat /etc/passwd | egrep [0-9]{3}-[0-9]{3}-[0-9]{4}
Easy syntax for using basic GNU regular expressions
The above example extracts lines that contain a phone number

Question: Why use EGrep when the exact same command works with Grep?
Answer: Depending on the flavor of UNIX you are using , Grep may behave differently. EGrep should always work the same. For example: While Linux's Grep allows for GNU regular expressions, AIX's Grep does not.

AWK: "Aho, Weinberger, & Kernighan" for trivia buffs
Very powerful beyond the scope of this discussion; I like to use it to extract certain words out of a line:

Example: cat /etc/passwd | awk -F: '{print $1}'
This prints out the first column in a colon-delimited file (such as /etc/passwd)

SED: Stream Editor
This is a great command for advanced text-parsing and manipulation. I'd recommend keeping a book or web tutorial handy for this- it's way too dense for this discussion.
See http://www.grymoire.com/Unix/Sed.html for some additional background on this tool.

Perl: You can use this extremely powerful programming language to parse text. Complete scripts can be used as well as the 'oneliner' variety:

Example: cat /etc/passwd | perl -ne 'split(":"); print "$_[0]\n";'
This prints out the first column in a colon-delimited file (same as the AWK example)

Welcome to Enterprise UNIX

2007-11-09T10:01:00.001-05:00

Welcome to my Blog! My name is John Muirhead-Gould. I am an Engineer Manager at JP Morgan Chase.

My team runs about 400 UNIX servers running AIX, Solaris, HPUX, and Linux (Red Hat Enterprise Linux). We work with a variety of high-end technologies in coordination with the overall operating systems: EMC Symmetrix & Clariion SAN, Veritas Netbackup & IBM TSM, and a variety of databases (DB2, Sybase, Oracle).

This blog is intended to provide a resource on how to manage a large, heterogeneous UNIX environment. I expect to cover some high-level/architectural topics as well as discussing some commands and techniques that compose our bread-and-butter of doing our jobs.