Workload Inspector (WiT) performance anomaly detection Tool across GPUs 
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. This PR was closed because it has been inactive for 7 days since being marked as stale.
# What does this PR do ?

Workload analysis tool for anomaly detection and perf analysis. Tool uses Nsys trace captures. 
PR takes Nsys profiling changes as a reference.
 
Provides diagnostic information at per GPU level by analysis of Nsys traces.
  - Compute vs. exposed communication. 
  - Communication overlap at each collective and stream level. 
  - Slow kernels causing training bottlenecks. 

```
Kernel time per iteration: 401.182 (ms)
Kernel instances per iteration: 1006
 Kernel Statistic:
 Class            TimePerIter(ms)  InstancePerIter  TimePerIter(%)   TotalTime(ms)    TotalInstance  TotalTime(%)
 others                    25.68             5513            2.90         256.79            55254          2.90
 bf16_gemm                231.65             21236           53.99        2316.48           212400         53.99
 layernorm                  4.81             2808            0.80          48.12            28080          0.80
 reduce                     2.95             1112            0.62          29.53            11148          0.62
 userbuffer                 39.42            31332           23.96         394.18           313344         23.96
 elementwise_kernel         88.76            29088            6.66         887.55           291128          6.66
 attention                 7.83             1841             10.92          78.32            18432         10.92

NCCL time per iteration: 285.951 (ms)
NCCL overlapped comm time per iteration: 175.113 (ms)
NCCL Overlap Statistics:
Class                  TimePerIter(ms)    OverlapPerIter(ms) Overlap(ms)        OverlapPerIter(%)
AllGather                 109.85              71.50             714.98            65.09
ReduceScatter             134.08             103.61            1036.15            77.28

Per NCCL stream Overlap Statistics:
Class                               Stream ID        Grid Size        TimePerIter(ms)    OverlapPerIter(ms) Overlap(ms)        OverlapPerIter(%)    TotalTime(%)
ReduceScatter                       16               6                          20.37              0.00              0.00                0.00          3.53
ReduceScatter                       16               3                          10.48              0.00              0.00                0.00          0.02
AllGather                                10               4                          81.74             32.62            50.48            8.24           19.74
AllGather                                46               2                        35.45             26.84            67.37            14.97           25.95
AllGather                               22               1                           0.05              0.00              0.00                 0.00          0.00

GPU bubbles per iteration: 0.04%
Multiple NCCL streams overlap per iteration: 0.00%

```

# Changelog 

- Added a Workload Inspector Lightening callback [nemo/lightning/pytorch/callbacks/workload_inspector.py](https://github.com/NVIDIA/NeMo/commit/df4887aead1210b94c2edb254e4c05c996f6749e#diff-dd4d5723656416f43d36613d894dcb66a1547cd6de072137d39f118b038838b2)

- WorkloadInspectorPlugin added to enable the tool [nemo/lightning/run/plugins.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-2b4c9a34429f8078889e60cccd2e2b6759100659c2afaa4ebbb493eb97704615)

- Added WiT callback to [nemo/lightning/pytorch/callbacks/__init__.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-c9e02dfa089e3aacf2f24c451f7af596b1573f22d0e4f218511aca881f6dd5e4)

- WiT enabling changes in Interface for Pytorch-lightning based NeMo models [‎nemo/core/classes/modelPT.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-4b936f7283de92c1563db15e6e966eea3994139ff3ff18823b498592e060a75c) 

- Invoke WiT plugin in [tests/lightning/test_nemo_run.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-65cf660991d9d1586bea1caa565d4f8abdf348a4e59432e66d064d79858b38b9)

- WiT enabling steps in various Megatron based models:
[‎nemo/collections/multimodal/models/vision_language_foundation/clip/megatron_clip_models.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-db5f2e6c1e3156d33d2e5afa097b7d425e20c07e021f79e43d9916b993027c48)

[‎nemo/collections/nlp/models/language_modeling/megatron_bert_model.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-d496e911eb06fca3128061957d326a83ba0a603ca8482ad292aee3851f49bf61)

[nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py]
(https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-d84a79499285d34da7d19099472502cb4e7c1b1672832488268066c9c7df852d)

[nemo/collections/vision/models/megatron_vit_classification_models.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-791dbc321fe2f4f37e07d17dd694d7a31b2b849510841958226c6d045b529035)

- Test cases for WiT 
[‎tests/lightning/pytorch/callbacks/test_workload_inspector.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-4b1ecd4d173033a01acb4e7389e7a6f043bb95916f063720ce9dc9ec3fe34319)

[‎tests/collections/nlp/test_falcon_model.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-ca09720d0bccee0bd8699c8342d3ba78b427c2b70f8a9387dbf5e8b21d435901)

# Usage

Workload Inspector (WiT) uses NVIDIA Nsight profiling and CUDA Profiler Start/Stop APIs to detect anomalies in training applications. 

```
 # Workload Inspector profiling options in yaml file
  wit_profile:
    enabled: False
    start_step: 10  # Global batch to start profiling
    end_step: 10 # Global batch to end profiling
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
